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USE AND USERS OF DIGITAL RESOURCES: 

A FOCUS ON UNDERGRADUATE EDUCATION IN THE 
HUMANITIES AND SOCIAL SCIENCES 

Diane Harley et al., 

Center for Studies in Higher Education, University of California, Berkeley 
http://digitalresourcestudy.berkeley.edu 

EXECUTIVE SUMMARY 



Introduction 

A "build it and they will come" approach to many university digitization initiatives has 
precluded systematic investigations of the demand for these resources. Those who fund and 
develop digital resources have identified the general lack of knowledge about the level and 
quality of their use in educational settings as pressing concerns. 

The purpose of our research was to map the universe of digital resources available to 
undergraduate educators in a subset of users in the humanities and social sciences (H/SS), and 
to examine how understanding use and users can benefit the integration of these resources into 
undergraduate teaching. 

Why study users? There are myriad reasons cited for undertaking and conducting user studies. 
They may range from product design and usability testing, to policing web sites, to facilitating 
policy and investment decisions. For our purposes there were three interrelated rationales for 
conducting the present research: (1) addressing questions of strategic planning and investments 
in digital resource provision and use, (2) identifying the special needs of the humanities and 
social sciences, particularly as they relate to the future of liberal education in a digital age, and 
(3) sharing effective strategies for understanding the array of uses and users across a wide 
variety of educational digital resource initiatives. 

Overview of methods 

Our specific approach was to employ multiple methods and empirical data to investigate how 
and if available digital resources are being used in undergraduate teaching environments. 

It is important to note that our definition of digital resources is intentionally broad and includes 
rich media objects (e.g., maps, video, images, etc.) as well as text. These digital resources may 
reside in or outside of digital libraries and include those developed by individual scholars and 
by other entities. 

Results 

There are three broad areas to our work reported here: 

1) A literature review and discussions with various stakeholders to provide a scan of the 
digital resource universe, and where the user fits into that universe 
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2) Surveys of faculty at various types of California higher education institutions as well as 
subscribers to humanities and social sciences listservs, and the testing of the 
combination of transaction log analyses and online surveys as methods for 
understanding users of specific web resources 

3) Interviews of site owners and the organization of a symposium to explore how 
gathering comparative user and non-user data across a variety of digital resources might 
be achieved 

1. Understanding the humanities and social science digital resource landscape 

Through (1) a literature review and (2) targeted discussions with those who create and 
disseminate digital resources, we assessed the landscape of user studies that target digital 
resources available to H/SS faculty in undergraduate settings. 

Our literature review encompassed the following broad areas: 

1) Humanities and technology; 

2) Real and virtual classrooms; 

3) Undergraduate education reform; 

4) Information literacy initiatives; 

5) Tools for instructional use of digital resources; and 

6) Specific research on digital resource users (electronic resource/ digital library use 
studies; cultural heritage research; site-specific user studies; image-service studies; and 
complex new media such as interactive video, GIS, 3D applications, games, and social 
software). 

The literature review reinforced our early assessment of the state of the landscape. We 
discovered that a description of this space entails complicated definitions about, and analyses 
of, (1) the scope, variety, and origins of the available rich media resources, (2) how the resources 
are actually used (or not used), and (3) the variation that exists between and among a diverse 
group of "users" and "owners" (and a recognition that users and owners are often embodied in 
the same person). 

There is an especially complex set of stakeholder interests and agendas when it comes to 
defining the value of user studies. There are policy makers and administrators who oversee 
educational reform or digital library efforts, developers who create resources, and technicians 
or designers who develop tools for the integration of resources into undergraduate settings. 

This diversity of perspectives and agendas complicates the understanding how an exceptionally 
diverse set of digital resources is actually used. 

Creating a Resource Typology: 

We convened a number of groups to discuss the digital resource provider or site owner 
perspective, and to tackle the problem of common vocabulary. But there are challenges to the 
construction of such a vocabulary, not least of which is that digital resources of all kinds are 
proliferating in many different environments and are created by many different kinds of 
developers. 
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We began creating a typology by simply describing resource types (e.g., curriculum, video, 
maps, electronic journals, etc.) based on actual faculty discussion group data (below), thereby 
generating a useful map for describing digital resources from a faculty perspective. We refined 
the typology so that resource characteristics are gathered around "centers of value." These 
centers of value (e.g., what and how content is represented, how can it be found, etc.) can 
function as broad yet significant guiding principles, with considerable strengths in describing 
many kinds of digital resources. 

What we soon discovered, however, is that users, when compared to resource providers, often 
use a different level of granularity in defining a resource (e.g., whether they can find on the web 
a format, a photo, a picture, or a passage). Furthermore, categories of users often comprise 
diverse individuals with varying and idiosyncratic needs, perceptions, and ways of finding and 
utilizing digital resources. 

The set of roles under the designation "owner," (and the individuals in those roles) ordinarily 
have different interests, values, and, especially, different levels of access to traces of user 
behavior. A colleague suggested the following distinction of these roles: 

• Aggregators, who select which digital resources are to be available in what 
combinations, and try to bring them to the attention of users 

• Developers of tools, who shape user interactions, export mechanisms, and access paths 

• Content creators and owners, who conceive, assemble, describe, and digitize content 

2A. Faculty discussion groups and online faculty survey 

To determine how, how much, and even if unrestricted digital resources are being used in 
targeted H/SS teaching and learning contexts among diverse higher education communities, 
we conducted discussion groups and a survey of potential users (faculty, teaching assistants, 
and students) from UCs, liberal arts colleges, and community colleges in California. 

Faculty discussion groups: In October and November of 2003, we hosted four sessions of 
discussion groups with thirty-one instructors from three institutions. The discussions formed 
the basis for the development and creation of the faculty survey. We asked variants of four 
basic questions in our discussions: 

• What digital collections/ resources do you use in undergraduate teaching? 

• How do you use them in your teaching? 

• What obstacles to use do you encounter? 

• What would you like to do with digital collections/ resources in a perfect world? 

The discussion groups are summarized in a separate publication that can be found at: 
http:/ / digitalresourcestudy.berkeley.edu/ pdf/faculty _discussion_group_june05.pdf 

Faculty survey: We conducted a survey of approximately 4,500 faculty from specific disciplines 
at a stratified random sample of community colleges. University of California campuses, and 
liberal arts colleges in California; the survey was administered both online and on paper. We 
received 831 valid responses (a response rate of 19%). A follow-up telephone survey of selected 
non-responders found no convincing evidence of response bias in the survey. We also 
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conducted a second, parallel survey of instructors from a broader range of institutions, 
disciplines, and geographic areas, recruited through online discussion groups; we received 452 
responses. The results from this second survey corresponded closely with the main faculty 
survey on most dimensions. 

The survey questionnaire focused on eight main domains: 

• Teaching background 

• Types and sources of digital resources used 

• Personal digital collections 

• How digital resources are used in teaching 

• Motivations for using digital resources 

• Motivations for not using digital resources 

• Barriers and frustrations 

• Support and assistance 

Results of the survey reinforced our early impressions from discussion groups. There is a broad 
spectrum of user types, ranging from the non-user, to the inexperienced, novice user, to the 
highly proficient and advanced user of digital resources. Non-users were themselves diverse. 
They included those who were passionately opposed to the use of technologies in their 
classroom for a variety of valid pedagogical reasons (e.g., these technologies cannot substitute 
for preferred teaching approaches). Non-users also included self-described enthusiasts 
frustrated by technical and non-technical barriers, and those simply without time to think 
about, let alone use, technology in teaching. The degree to which personal teaching style and 
philosophy influence use was striking. 

Respondents used a wide range of resource types for a variety of reasons. Images and visual 
materials were the most frequently used resource, and were often used for classroom 
presentation or posting on the web. Google-type searches were the most frequent way in which 
faculty found resources. A faculty member's own "collection" of digital resources was the 
second most frequent source of material. 

Faculty used digital resources to improve their students' learning, to integrate primary source 
materials into their teaching, or to include materials or teaching methods that would otherwise 
be unavailable. Some said they used digital resources because it was expected by their students 
or their colleagues. 

The foremost reason for not using digital resources was that they simply did not support 
faculty's current teaching approaches. Lack of time was a major constraint, regardless of 
institution. We can say that it was not at all easy for most of our respondents to use the 
plethora of digital resources available to them. Faculty — including those active and enthusiastic 
in their use of digital resources — identified many obstacles to using these resources for teaching 
including how to find, manage, maintain, and reuse them in new contexts. One of the most- 
cited obstacles to the effective use of digital resources was the availability, reliability, and 
expense of the necessary equipment, both in the classroom and for personal use. Almost all 
faculty need support for a variety of tasks. Both novices and advanced users face challenges 
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when integrating digital resources into their teaching, but they experience somewhat different 
needs and barriers; thus, support systems that are helpful to one group may not be for another. 

Responses also emphasized the importance of personal digital collections in faculty work 
practices. It is important to emphasize that many faculty want to build their own reaggregated 
resources by using their own materials and then mixing them with resources they have 
collected along the way. How to manage the multitude of available resources and integrate 
them into teaching practice is a major hurdle. Although there may be an array of tools available 
to faculty for collecting, developing, and managing resources, the efficacy and interoperability 
of these tools for the immediate tasks that faculty need supported often fall short. 

An analysis by discipline revealed variation among scholarly fields. Faculty who use texts 
extensively depend on different kinds of sources for different pedagogical goals than faculty in 
art, architecture, history, and anthropology, who rely more heavily on images. Faculty in 
political science were the heaviest users of data sets, and faculty who teach writing had special 
needs around information literacy and the use of reference materials. Not only do different 
disciplines require different types of resources, they use them in different ways and for different 
reasons. 

When the data were analyzed by age, the oldest instructors (age 62 and up) were the lightest 
users. A multiple regression analysis demonstrated, however, that age alone is a very weak 
predictor of a person's overall level of digital resource use. Regression and path analysis further 
showed that individual characteristics have a greater effect on a person's total level of digital 
resource use than do institutional, disciplinary, or demographic characteristics. 

2B. Investigation of transaction log analysis and online site surveys 

We set out to evaluate two methods for tracking site usage. Server transaction log analysis 
(TLA) is a widely used method for studying the usage of particular websites. Many sites also 
use web-based surveys, either alone or in combination with transaction log analysis, to learn 
more about their users. The lack of consensus on how best to implement these methods and 
report on the results, however, makes it difficult to interpret statistics for different sites and to 
compare one site's findings with another. 

We explored the benefits and challenges of these two user research methods by pilot testing 
their efficacy on two local websites. Our charge was not primarily to generalize our findings 
about users of these specific sites; rather, we were interested in the efficacy and efficiency of 
TLA and online surveys as methods for understanding use generally. 

Results from the analysis of both survey and TLA indicated that the overwhelming majority of 
site users on our two test sites were irregular or occasional, rather than regular, users. An 
analysis of search engine queries that led to each site provided some insight into users' goals 
and mindsets. The design of each site also affected both the analysis and the usage itself. 

Given our knowledge of how site owners report out on user behavior, we were interested in 
exploring how representative online site survey results actually are. Specifically, combining 
online surveys with transaction log analysis of the same site during the same time period 
creates opportunities for measuring the survey's response rate and for estimating response bias. 
To assess whether the survey respondents were representative, we compared the observed 
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browsing behaviors of those who did to those who did not respond to the survey. For both 
sites, results indicated that the users who responded to the survey were noticeably different 
from the typical site user: survey respondents used each site much more frequently, and each 
session was longer and more in-depth. Therefore we concluded that the online site surveys 
suffered from response bias, and the respondents were a non-representative sample. Although 
the tests we conducted helped to clarify the specific techniques and metrics that provide the 
most useful insights into site usage, the tests also provided a vivid demonstration of some of the 
challenges and pitfalls in performing user research and interpreting results. Based on these 
results, we advise caution in generalizing from online site survey results to the whole 
population of site users. 

3. Why study users? Summary of digital resource provider and researcher interviews 
and meeting 

In keeping with our multi-pronged approach to understanding users of digital resources, we 
devoted considerable time over the course of the project to talking with digital resource 
providers about why they studied users, what they knew about users, and what more they 
would like to learn. In addition to conducting formal interviews, we convened a group of 
resource providers, funders, and user researchers for a two-day meeting in May 2005. 

Interviews with digital resource providers: Interviews with thirteen digital resource providers 
and two other stakeholders in the field underscored the diversity of projects, tools, and services 
available to the H/SS community and the difficulty of making comparisons among them. Our 
sample included sites that provided educational online resources and that had at least some 
freely available resources. The goals of these interviews were: (1) to test our initial sort of 
digital resource characteristics, (2) to collect opinions on the importance of user research to 
digital resource providers, and (3) to determine if certain factors and their attendant digital 
resource characteristics (e.g., histories, funding models, architectures, etc.) are associated with 
successful strategies for integrating an understanding of users into development and 
maintenance activities. Where possible, data on cost and collaborative development strategies 
were collected. 

The interview analyses suggest that there were no common terms, metrics, methods, or values 
for defining use or users among the targeted projects. One common theme among digital 
resource providers was the desire to measure how and for what purpose materials were being 
used once accessed; few providers, if any, however, had concrete plans for undertaking this 
measurement in a systematic way. 

Many digital resource providers targeted faculty as their primary audience. Several sites, 
however, are exploring expansion to new audiences either through targeted planning or in a 
more serendipitous fashion. Our research revealed that community building is important to 
digital resource providers, and many are exploring tools to enable the development or support 
of user "communities." Some have also suggested that community contributions can hold a key 
to sustainability challenges. 

Sustainability for the initiatives we researched is a pressing, if elusive, question for most sites. 
Formal agreements or plans to determine long-term financial, technical, and organizational 
sustainability are practically nonexistent. Success and value were slippery topics, though it is 
apparent that high-quality projects often bring advantages to their institution's resident 
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students and faculty, and can also bring some level of prestige to the larger organization. This 
recognition of value by the sponsoring institution provides a potential route to long-term 
support and funding. 

Site Owner and User Researcher Meeting: In May 2005, we convened sixteen experts for a 
discussion of "online educational resources" (OER) to explore how and if questions about user 
behavior are tightly linked to questions of policy and planning. A majority of participants had 
been interviewed before the meeting (above). Our discussions covered four broad topics: 

• Codifying content and contexts 

• What do we want to know about users? How do we find out? 

• Users, user demand, and sustainability 

• What are the larger research questions and agendas that need to be addressed? 

The participants represented a variety of perspectives in the field of online educational 
resources. Their collective expertise included production and delivery of online educational 
resources, delivery of continuing education, user research, and foundation funding. The 
following organizations were represented: 

• Curricula: CMU OLI, Connexions, MERLOT, UC Irvine, MIT OCW 1 

• Digital libraries: JSTOR, ARTstor, NSDL 2 

• Tools and reuse: Carnegie Foundation, MIT, IKSME 3 

• Foundation and society perspectives: ACLS, Hewlett Foundation 4 

Finding a common framework: Codifying categories of content, users, uses, and user 
studies 

Comparing data among OER projects poses a significant challenge to those who conduct or rely 
on user studies for decision-making. A valuable goal, according to many participants, would be 
for the OER community to articulate general principles and standards for user studies, and for 
sharing usage statistics and results. 

Meeting participants agreed that any conversation about users needs to first establish a 
common framework and vocabulary to ensure successful comparisons among projects. 
Similarly, when we discuss results of user studies across projects, it is important to codify terms 
and methods, and to understand the full range and purposes of research methods and tools 
available for evaluating users. The practicalities of conducting user studies present some 
obstacles, however: high-quality research is resource intensive, and in-house evaluations can 
result in a "self-fulfilling prophecy," where studies frequently favor relationships and products 
that already exist. Another stumbling block to collecting good data is the difficulty of 
understanding the full range of an OER site's users (e.g., users who do not register, users who 



1 http://www.cmu.edu/oli, http:/ /cnx. rice.edu, http://www.merlot.org, http://learn.uci.edu, http:/ /ocw.mit.edu 

2 http://www.jstor.org, http:/ /www. artstor.org, http://www.nsdl.org 

3 http://www.carnegiefoundation.org/KML, http://www.iskme.org 

4 http://www.acls.org, http://www.hewlett.org 



http:/ / digitalresourcestudy.berkeley.edu 



1-7 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



do not log in from recognizable institutions, the difficulty of interpreting transaction log data, 
etc.). 

Sustainability 

Definitions of value and approaches to sustainability vary according to each OER's context and 
goals. It was argued that the only way to understand the value of OER— for individuals, 
communities, and institutions — is to measure its impact and its outcomes. It was agreed that 
disaggregating the ingredients of sustainability was essential. Four types of sustainability were 
subsequently identified: curricular, technical/ infrastructural, organizational, and financial. 

Curricular sustainability: It was agreed among the participants that creating and sustaining 
high quality curricular resources can be costly. Real concerns were voiced about the potential 
for rampant propagation of misinformation and poor quality educational materials on the 
Internet. While producers can actively control quality by strictly enforcing their own 
pedagogical and production standards, doing so can make the material difficult to reuse outside 
the context originally envisioned by the producers. Alternatively, the user community itself can 
take the place of institutional or individual authority over quality, although fears about 
diminution of quality are an especial concern among content experts with this model 
(enthusiasm about Wikipedia in some circles notwithstanding). 

The development of user communities among OER sites, as a measure of curricular 
sustainability, was discussed at length. Participants recognized ongoing problems with 
community reuse. Currently there is no common set of standard tools or practices to help 
achieve interactive community on a large scale in educational contexts, though emerging social 
computing models were again noted by some as promising. 

Unintended and informal users. One question that arose was whether OER sites could or should 
adapt their content or services to unintended users. To some participants, unintended use is an 
opportunity for creative reuse, while many believed that an OER site should not or could not 
change course to serve an unintended audience. How a site accommodates unintended use 
may require a complicated calculus taking into account the site's mission, scope, financial 
model, desired impact, quality control, and targeted constituencies. It was agreed that studying 
unintended users is exceptionally difficult, and that accommodating them in site development 
can incur potentially significant costs. 

Technical/infrastructural sustainability: It was argued that OERs, and especially open-access 
OERs, need a common place where they can be reliably housed, organized, searched, and 
preserved, perhaps in one or more centralized OER repositories. How a centralized repository 
would be organized was open to debate, however, and several possible solutions were 
discussed. Several participants agreed that federating searching among all OER sites would be 
desirable and most "user friendly." 

Organizational sustainability: Organizational value is related to how OER fits into the 
organization that supports it. To what degree does the host institution value the OER site, and 
to what degree does the site's value drive institutional support? In many cases there is an ad hoc 
approach in which a faculty member cobbles together local support. If he/ she leaves the 
institution or runs out of funding, the OER can potentially be compromised. Long-term 
commitment for OER is often unclear. 
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Financial sustainability: At this stage, many OERs depend on a mix of institutional, 
foundation, and corporate funding, and few have concrete plans for financial sustainability. 
Various "business" and financial models were discussed, including endowment models, 
subscription models, and others. There was discussion about the hesitation in academic circles 
to endorse the concepts of business models, market research, and sustainability. For those 
OERs that wish to remain non-commercial entities, a combination of foundation, institutional, 
and corporate funding nonetheless remains the only source of financial sustainability. The 
questions remain: how do we define and measure "critical mass" relative to a specific OER, and 
how do we measure, then demonstrate, successful outcomes with funders? 

Imagining a research agenda 

We concluded our meeting with a hypothetical scenario in which participants were asked to 
brainstorm how research funding should be spent on understanding OER users. Participants 
agreed that all studies should be coordinated to use a similar set of terms and techniques, so 
that findings can be shared more effectively and made generally applicable. Multiple research 
topics were suggested, but they converged on one primary and one secondary research priority: 
faculty and self-learners, respectively. 
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INTRODUCTION AND RATIONALE FOR THE PROJECT 



There is consensus among many scholars, developers, and the public that high-quality 
knowledge should be freely available where economically feasible. Almost every American 
research university campus, including those in the University of California (UC) system, has 
made significant investments in digitizing its intellectual and cultural resources and making 
them available to faculty, students, and the general public. However, we have little empirical 
data about how these resources are being used by the originating campus or by other 
institutions for educational activities. The general lack of knowledge about level and quality of 
use of "unrestricted," or free, resources has been identified as a pressing concern by those who 
fund, use, and develop these types of resources. The "build it and they will come" approach 
has resulted in a widely acknowledged supply-driven movement. For example, after providing 
millions of dollars for the creation of digital libraries, NSF and JISC (the U.K. equivalent of 
NSF ) 5 are both concerned about the low level of use of available digital resources among the 
teaching faculty of our institutions. Those who fund open educational resource initiatives, such 
as the Hewlett Foundation, are interested in users and how understanding them will provide 
insights into the sustainability of the significant activities they fund in this space. 

Our motivations for this study were driven by three interrelated rationales: strategic planning 
and investments, focus on the humanities and social sciences, and consolidation of effective 
strategies for understanding use. 

1. Strategic planning and investments. Strategic investment decisions by funders and 
institutions will undoubtedly hang on the question of how to pay for the significant costs of 
digital resource production and maintenance . 6 The question of cost becomes more pressing in 
an era of shrinking institutional budgets and deflated expectations of consumer markets for 
digital curricular materials . 7 

We know very little about how digital resources, such as those produced at research 
universities, are actually being used by the different tiers of higher education institutions both 
in the U.S. and abroad. There is an implicit assumption that faculty at a variety of institutions 
import digital content to enhance their undergraduate teaching. We simply do not know, 
however, if such importation occurs on a measurable scale. And if it doesn't, why not? This 
question is of particular importance in California, where there is a presumption that digital 
assets will flow from the public UC research university system to institutions with fewer 
resources, such as high schools, community colleges, state universities, and the new campus at 
UC Merced. 

What are the costs to institutions in creating and maintaining these digital assets? That question 
is perhaps impossible to answer given the mosaic of development and funding models that 
exist for any one set of assets. In many cases, funding sources are cobbled together from a 
variety of institutional and foundation budgets. There are the often unpredictable, ongoing 
costs of maintenance and updating. And what of the significant costs incurred by digital 
resource developers to simultaneously meet the needs of audiences that range from scholars to 
school children, both internationally and domestically? 



5 http://www.nsf.gov, http://www.jisc.ac.uk 

6 In reference to digital libraries in particular, see Smith (2001) and Smith et al. (2004). 

7 See Matkin (2002) and Cushman (2002). 
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In addition to institution-sponsored resources, there is the growing mass of "educational" 
digitized rich media objects created by individual scholars and others. What is their value, and 
who will maintain and preserve them? As decisions are made about financing resource design 
and scope, an understanding of the level and type of use of these, as well as more "organized" 
resources, will be needed in strategic planning. 

We believe that a focus on understanding the use of both unrestricted and restricted digital 
resources can shed some light on whether the investments in production and distribution of 
"free" resources is warranted by market demand relative to undergraduate teaching (and other) 
contexts. 

2. Focus on humanities and social sciences. Within the academy, there is an abundance of good 
models for integrating online materials in science and technical courses such as chemistry, 
physics, biology, and computer science (see, for example, Twigg, 2003 and Fisher and Nygren, 

2002) . The application of technical solutions to undergraduate teaching in the humanities and 
social sciences (H/SS), however, has been more elusive and less robustly funded. In fact, some 
have argued that technical and professional courses, where there is a heavier reliance on 
codified knowledge, may be more amenable to technological interventions (e.g.. Trow, 1997). 

At this time there appears to be a paucity of literature that has systematically examined this 
problem, especially as it relates to the integration of non-text, non-library resources. 

Our own experience and discussions with faculty suggest that successful integration of 
technology in H/SS teaching may be stymied in courses that rely heavily on primary source 
material and significant verbal and written interaction among participants. We have identified a 
number of factors that might prevent routine integration of rich digital archives in the 
classroom. They include constraints on faculty time, lack of support structures, and difficulties 
with finding, analyzing, and customizing the abundance of online material available. Without 
doubt the most important hurdle may be that each faculty member has precise and personal 
ideas about how he/ she teaches a course and what kinds of primary source materials are 
useful; not every faculty member will teach U.S. history, Chinese literature, or foreign policy in 
quite the same way or use the same primary source material. This fact is in marked contrast to 
many undergraduate courses in scientific, technical, and vocational fields, where scope and 
sequence are more tightly constrained by the necessities of moving students in a stepwise 
fashion through a series of linked requirements. Perhaps most importantly, it cannot be 
ignored that many faculty do quite well teaching without technology and, given the barriers to 
use, would rather avoid the associated headaches. 

Our research targets a better understanding of the variation in user and non-user behavior 
according to discipline and institution, by focusing on a subset of the HE disciplinary 
landscape. The humanities and social sciences are not a monolith, nor are user types. Studies 
that have focused on the use of library materials indicate that patterns of use are influenced by 
the type of institution and discipline. For example, "survey respondents in the liberal arts 
colleges and in the biological sciences and the arts and humanities seem to rely on the library 
and its functions and its services more than their peers in other disciplines do." (Friedlander, 

2003) . Greenstein and Thorin (2002) point out that an additional obstacle to the use of digital 
library materials in undergraduate teaching is "the near universal deployment of instructional 
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technologies that do not integrate with the digital library" and the creation of digital collections 
that rarely address instructional needs. 8 

Finally, we believe that understanding the technology needs of the humanities and social 
science community has particular relevance to the future of liberal arts or general education 
delivery and the increasingly vocationalization of higher education (Rothblatt, 2003; Smelser 
and Schudsen, 2004). Any discussion about the future of liberal, or general education, cannot 
ignore the new cohort of "always on" students that is poised to enter higher education 
institutions. Future planning will be confounded by the fact that we simply do not understand 
enough about these students who will have been weaned on peer-to-peer file swapping, Google 
searches, MySpace, and wireless instant messaging (Harley, 2002; Oblinger and Oblinger, 2005; 
Kvavik and Caruso, 2005). What expectations will these students have about their learning 
environments and the nature of scholarship? How will institutions respond to students who 
may have non-traditional concepts of time and space in scholarship? 

Indeed, the very future of liberal arts is of concern on a global scale. We do not know how 
many students will eschew traditional liberal arts curricula for the immediate economic benefits 
of management and technology education. It is probably safe to assume that as new online 
education providers proliferate and consolidate, the range of educational choices available to 
students will increase, and many mature students will forsake a traditional four-year residential 
college experience for certification and part-time degree programs (Harley, 2002). In a white 
paper prepared for a meeting sponsored by the Center for Studies in Higher Education (UCB) 
on the regulation of cross-border e-learning, Kumar et al. (2005) posit: 

One consequence of greater cross-border e-learning may be a narrowing of subject 
concentration. Seeking to maximize income and meet mainstream demand, much cross- 
border higher education has concentrated on strongly career-oriented provision (notably 
business, IT, healthcare, and education). The broader subject mix typical of the 
comprehensive university has not been widely replicated. With notable exceptions, there 
is little evidence of constructive relationships between cross-border providers and host 
governments, and thus little apparent connection between provision and perceived 
national skills needs. By focusing on more lucrative programs, cross-border delivery may 
cut across domestic higher education in negative and unpredictable ways. 

The ACLS Commission on Cyberinfrastructure for the Humanities and Social Sciences 9 (2005) 
lays out in great detail the "grand challenge" of harnessing the potential of information and 
communications networks to serve the special and diverse scholarly needs of humanists and 
social scientists. It is our contention that solutions to those scholarly needs and related 
challenges will ultimately benefit the future of teaching and learning in the humanities and 
social sciences, and the challenges are worth addressing sooner rather than later. 

3. Consolidation of effective strategies for understanding use. It is worth noting at the outset 
that the terms "digital collections" or "digital resources" are often used synonymously with 
"digital library collections." This study intentionally focuses on rich-media digital resources 
that may reside in or out of libraries. The variety of digital resources is extensive. Nationally 
and internationally, unrestricted resources range from ambitious attempts to put up course web 



8 Also see the recent Ithaka faculty survey of electronic resources for a discussion of disciplinary variations in use 
(Schonfeld and Guthrie, 2004). 

9 http://www.acls.org/cyberinfrastructure/cyber.htm 
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pages or whole courses (e.g., MIT OCW, World Lecture Hall, Monterey Institute for Technology 
and Education), to discipline-specific course materials (e.g., LON-CAPA, Connexions), to 
clearinghouses of individual learning objects (e.g., MERLOT), to digital library/ museum 
collections (e.g., CDL/MOAC, Harvard Open Collections Program), to collections assembled by 
individual scholars. 10 Moreover, the few available user studies on this vast array of resources 
are themselves fragmented by purpose, method, and context. 

To date, there simply has been no coordinated conversation about user research that could 
apply across the many types of available digital resources and their sources. Lack of a clear 
picture about users makes coordination (of user methods, findings, business models, strategic 
planning) across projects challenging. One focus of our research is to ask whether it is possible, 
or even desirable, to have projects share methods and results. Is there too much variation 
among projects — in terms of content, objectives, targeted users, funding models, and need for 
understanding users — to attempt to coordinate knowledge about users (and methods for 
studying them)? 

What is the overall value of "user" studies? How can we begin to assess overall user demand, 
and what analytic methods are useful for the various phases of decision-making (e.g., start-up, 
site design, dissemination, maintenance, scaling, new audiences, etc.)? For example, usability 
studies or testing of pedagogical applications in the classroom are clearly useful for site and 
content design, but they will not yield the kinds of data one needs when making decisions 
about initiating a new project, developing funding models, or assessing/ targeting new 
audiences. There are a number of very good studies of the former type. Unfortunately, they tell 
us only about relatively enthusiastic users of a particular brand of content, but nothing about 
whether that brand may be valued or useable by a wider potential audience operating in varied 
educational contexts. 

An additional issue regarding use studies and those who conduct them relates to the return on 
investment (ROI) of various types of user studies (explored by Hill et al, 1997). Not only is the 
issue of commensurate quality and quantity of information important for the developer, ROI is 
equally important for participants in user research. In what ways do investments in use studies 
actually translate to direct benefits to the user through simplifying their resource search, 
organization, and reuse needs? We have found that a time burden is placed on users who 
participate in research. What do they get in return and how do they directly benefit from such 
participation? Quite simply, can the cost of studying users in academic settings yield 
commensurate quality in the development of resources, tools, and systems? 



10 http:// web. austin.utexas.edu/wlh, http://www.montereyinstitute.org, http://www.lon-capa.org, 
http: / / www.bampfa.berkeley.edu/moac, http: / / ocp.hul.harvard.edu 
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RESEARCH GOALS AND METHODS 



There are three broad areas covered by this study: 

• Goal 1, which comprised a scan of the digital resource universe specific to the 
humanities and social sciences, and where the user fits into that universe; 

• Goal 2, which included surveys of faculty at various types of California higher education 
institutions as well as subscribers to humanities and social sciences listservs, as a way of 
determining what resources faculty use and why and how they use them. It also 
included an assessment of how effective the combination of transaction log analyses 
and online surveys are as methods for understanding users of specific web resources; 

• Goal 3, which included interviews with site owners and user researchers, and the 
organization of a symposium, to facilitate discussion and dissemination of simple, 
shareable, and cost-effective analytic models/ tools. A goal of the meeting was to explore 
how gathering comparative user and non-user data across a variety of digital resources, 
by a diversity of audiences, might be achieved. 

OVERVIEW OF THE METHODOLOGY 

This research project proceeded on parallel tracks. We utilized a suite of methods in our 
sampling of relevant populations, including unrestricted resource owners, faculty, and other 
users. More detailed descriptions of our various methodologies can be found in the relevant 
sections of this report. 

Our choice of methods was determined by the nature of our goals. We were not attempting to 
use our research either for product testing or for rigorous analysis of learning outcomes. Instead 
our goals were to provide (1) a relatively quick scan of use across a wide range of unrestricted 
digital resources by a variety of user types, and (2) a possible analytical model that could yield 
comparable data across a variety of digital resources. Given these goals, we used a combination 
of surveys, discussion groups, follow-up interviews, and transaction log analysis (TLA) to get a 
broad look at user and developer behavior. We have chosen to use these multiple methods and 
to triangulate results because our study is focusing on complex social settings and rapidly 
evolving technologies (e.g., faculty, random users, developers, technical systems). Marchionini 
(2000) and Gilliland-Swetland (1998) argue powerfully for such an approach in studies 
evaluating the integration of digital "repositories" into teaching/learning environments (see 
Marchionini also for a discussion of quantitative and qualitative methods, and information 
science research). 

For this project, we conducted background research to inform our data collection methods and 
developed a landscape of existing digital resource use. We selected methods to be 
complementary. The transaction log analysis gave us a general understanding of how useful 
this method is for establishing user profiles and for identifying patterns of user behavior based 
on geographic region and institutional affiliation. We tested online surveys to provide more 
detailed information from a self-selected subset of digital resource users to inform us about who 
they are, how they use the resources, and their opinions about online resources in general. 
Interviews and discussion groups provided information about targeted user groups, and 
elucidated reasons for non-use among specific disciplinary populations. 
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The scope and timeline of this study did not allow for a deep analysis of undergraduate student 
users. This is indeed an area ripe for study but outside our immediate purview, although we 
held a few focus groups with local undergraduate students and refer to our other work with 
this population of higher education users where appropriate. 

Some thoughts on user research methods 

We think it useful in a study such as ours, which is focused on users of online educational 
content and how to study them, to place our work within the wider context of user studies 
generally. A wide variety of research methodologies has been used for studying the use and 
users of online educational materials. To that end, a very brief review of the possible types of 
user studies follows. Several commonly discussed user research methods are described: 
surveys, transaction log analysis, observational studies, focus groups, and user testing. Some of 
these methods, such as surveys, focus groups, and observational or ethnographic methods, can 
address a variety of topics. Other methods, such as server transaction log analysis, are 
particularly relevant to studies of computer or Internet usage. 

The following is not intended as a literature review. We note references throughout the text for 
those interested in exploring this topic more deeply. Additionally, we have cited relevant 
methodological references when appropriate in specific sections of this entire report. 

a. Surveys 

Many types of surveys are used in social science research, including those distributed by mail, 
conducted by telephone, face-to-face, or even online. Surveys may be targeted at current users 
of a particular online resource or at a target group of potential users, such as students or 
educators. Surveys tend to be the methodology of choice for assessing users' needs, 
motivations, attitudes, and satisfaction levels, as well as self-reported behaviors (Rossi, Wright, 
and Anderson, 1983). Presser (2004) provides an excellent examination of methods for 
evaluating survey questionnaires. 

As web-based applications become widespread, more surveys are being administered in an 
online environment (Fricker and Schonlau, 2002). Online surveys can improve efficiency by 
automating data collection and analysis. Computerization may also enable more complex skip 
patterns and allow researchers to customize the questionnaire to the respondent's specific 
circumstances. Most of the considerations common to other survey modes still apply, however: 
instrument design, question validity, sampling technique, and response rate. (See also Evans 
and Mathur, 2005.) 

Online surveys encompass several different scenarios with very different research implications, 
and we should be careful not to conflate these situations. 

Surveys can be administered online as part of a traditional, well-developed survey 
methodology involving a defined population of interest, an explicit sampling method for 
generating a representative sample, a well thought-out recruitment strategy, carefully 
calculated response rates, and statistical estimates of the likelihood of response bias (Evans and 
Mathur, 2005; Fowler, 2002). As with offline surveys, issues of recruitment method and 
sampling are particularly important. (If recruitment is by email, for example, messages are 
likely to be ignored as spam.) 
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Increasingly, however, online surveys are posted on a website and made available to anyone. 
Such surveys may be part of another webpage, or may appear in a pop-up window. These 
surveys rarely have a defined population or sampling method; with no way of tracking those 
who do or don't complete the survey, it is often impossible to report a response rate or estimate 
response bias. These challenges may make it impossible to know if the resulting statistics have 
any value; such surveys are more of a "popularity contest", closer to the "American Idol" 
television show or baseball's All-Star Game balloting than to a real scientific research study. 

(See Goal 2B, below.) 

b. Transaction log analysis 

Transaction log analysis (TLA) takes advantage of the computerized log files that automatically 
record online accesses to any website. By analyzing these logs, one can determine a number of 
characteristics of the site's users and can summarize site use (Rosenstein, 2000; Kassim and 
Kochtanek, 2003). TLA is unusual among research methods in that it allows the researcher to 
measure the actual online behavior of a site's users, rather than relying on self reports. 

Although it can report a user's behavior with some accuracy, the method's weakness is its 
inability to report anything about the person actually performing those actions. The quality and 
detail of the TLA results can depend on a number of factors, including the design and 
architecture of the site, the use of identifying cookies, and the registration requirement of 
signing in. Although cookies and site registration may improve TLA's rigor, they raise 
important privacy and confidentiality concerns, particularly at non-profit educational 
institutions and libraries. Even in the best of circumstances, there will be gaps in TLA's 
effectiveness due to proxy servers, caching, blocked cookies, and the lack of accurate geographic 
information. (For more detail see page 5-2, below.) 

c. Observational studies 

Observational studies typically use ethnographic methods and involve passive observation of 
users (or potential users) performing their regular day-to-day activities in their own 
environments. An in-depth observational study can provide insights that are overlooked by 
other methodologies. In particular, such studies are ideal for capturing the details of users' true 
work processes (which often differ from self reports) and for seeing the way a tool of interest is 
used in combination with other tools. On the other hand, observational studies are usually 
time- and labor-intensive. Because they typically study only a small number of users, one must 
always question how representative the results are and the degree to which they can be 
generalized; the strategy for recruiting representative participants is particularly important. 

And users may somehow adjust their behavior due to the presence of the observer, skewing the 
results. See Weiss (1994) and Emerson et al. (1995) for excellent introductions to interviewing 
and ethnographic methods, respectively. 

d. Focus groups 

Like surveys or interviews, focus groups are a good method for exploring users' motivations, 
attitudes, thought processes, and satisfaction. Because the participants are usually few and not 
necessarily representative, the results should be considered suggestive rather than 
representative or comprehensive. Even if the participants are chosen to be representative, the 
results may not be, since the participants' comments can influence one another, spark new 
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ideas, and lead the conversation in a particular direction. Focus groups can be a valuable 
method for exploring the scope of a problem, examining particular issues in depth, or 
developing an early version of a survey questionnaire (see Gearin and Kahle, 2001). 

e. User testing 

User testing is typically performed in a specialized lab, instrumented to record exact keyboard 
and mouse movements, to videotape the participant (perhaps from several angles), and 
sometimes to track eye movements (to identify the exact location of a user's attention) (Nielsen, 
1994). Usually the participant is asked to complete several specific tasks or usage scenarios and 
may be asked to complete a short survey or interview before or after the test. Such tests tend to 
focus specifically on the product's usability rather than on its usefulness or its applicability to 
the users' real needs and environment. Because the controlled environment is artificial, 
participants may not behave normally. The prescribed usage scenarios are often somewhat 
artificial as well, and may not adequately match the users' actual situations. In addition, user 
testing can be very time- and labor-intensive and often requires a dedicated facility with 
specialized instrumentation. 

Choosing appropriate methods 

Different methods are appropriate for different phases of the product development cycle 
(Nielsen, 1994). User-needs research or "market" research should be conducted prior to the 
development of a new product or website. Rather than focusing on a particular site, user-needs 
research will typically involve the study of a particular target user group, to better understand 
their work practices and their needs; ideally, this research will direct the design and 
development process (as well as the decision about which systems should be developed). 

Useful methods for user-needs analysis and market research include surveys, focus groups, 
interviews, and observational studies; a successful research plan will require the careful 
identification of the target user group and the tasks or work practices of interest. 

The evaluation of an existing online resource is a different problem, and different methods may 
be appropriate. An existing system will presumably have an established user base with 
established work processes, making it possible to study the actual use of the system. Transaction 
log analysis and user surveys are probably the most useful methods for this situation, although 
interviews, focus groups, and user testing may also be appropriate. 

The evaluation of a newly designed or prototype system or website poses a particular challenge. 
Transaction log analysis will generally be unhelpful, since there will be no established user base 
or established work practices. Surveys and interviews will also fail to reveal much about the 
actual usage of the system, since users (or prospective users) will have no real experience using 
the system in a real-life context. In this situation, controlled user testing is probably the best 
research method; user testing involves an actual product or system, but can simulate the use 
environment with specific use scenarios. 

Well-designed user studies often employ multiple complementary research methods, as 
appropriate for the specific circumstances and study goals. 
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GOAL 1: UNDERSTANDING THE HUMANITIES/SOCIAL SCIENCE DIGITAL 
RESOURCE LANDSCAPE AND WHERE USERS FIT INTO IT. 

Primary goal: To define the universe of digital resources available to undergraduate educators 
in the humanities and social sciences and to examine how understanding use/ users can benefit 
the integration of digital resources into undergraduate teaching. Part of this goal was to 
facilitate the gathering of comparative use data across a wide variety of humanities/ social 
science (H/SS) digital resources. Activities directed toward this goal included creating a map of 
the universe of digital resources available to undergraduate educators and consolidating 
existing knowledge about the use and users of these resources. 

A. Defining the universe of digital resources and associated user studies 

Our first step was to conduct an analysis and overview of existing knowledge about use, and 
the methods for tracking use, of unrestricted digital resources. We have (1) reviewed and 
synthesized the existing literature on user studies relevant to unrestricted digital resources, 

(2) created a preliminary classification of select unrestricted digital resources in the humanities 
and social sciences, (3) through interviews, analyzed what developers / funders want to know 
about how (or if) unrestricted humanities / social science resources are being used in 
undergraduate teaching contexts, (4) analyzed what select developers/ funders are doing, and 
how much they are spending on user research, and (5) analyzed what developers know and 
their perspective on the best methods for measuring resource usage. 

The key challenges to pursuing an analysis and overview about use included: 

• Agreeing on how to define a collection of digital resources 

• Agreeing on a set of characteristics for certain types of resources as we developed the 
digital resource classification 

• Determining how to integrate the various perspectives of different types of site 
"owners" and users 

• Identifying those sites most likely to yield information that could be generalized to a 
wider universe of resources. 

1. Unrestricted digital resources: What are they? How can we organize them? 

Some background, both on our subject of study and our attempts at classification, is necessary 
from the start. 

The ACLS Commission on Cyberinfrastructure for Humanities and Social Sciences (2004) 
recently noted that digital objects and data in humanities digital resource collections tend to be 
far more complex and ambiguous than their counterparts in the sciences. Waters (2004) 
discusses factors contributing to the sustainability of digital scholarly resources. He notes the 
significant problem of the sometimes "disconnected jumble" of library digitization projects that 
do not connect across institutions. Smith (2003) looks at preservation and sustainability issues 
around "wholly new types of information resources, so novel that no common term except 
'digital objects' or 'sites' can describe them." We argue throughout this report that when non- 
library digital resources (a ubiquitous element in what most users desire from resources) are 
added to the mix, the impact on ease of use is obvious and profound. 
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Attempting to classify the mass of material available to users can be approached in myriad 
ways and presents many challenges, as we describe in some detail below. For example, a major 
study codified over thirty digital library aggregation services and non-library digital resources 
into clusters and described a typology for classifying specialized digital resources collections 
(e.g., American Memory, MERLOT, and Perseus 11 ) (Brogan, 2003). Although some useful 
definitions for classifying the ever-widening field of digital resources are developed, the study 
analyzes only those digital resources that adhere (to a large degree) to the Protocol for Metadata 
Harvesting of the Open Archives. 12 As we discuss below, there is a tendency for classification 
schemes to only focus on small corners of the universe (i.e., cultural heritage institutions or, 
more often, formal digital libraries). 

For our project, we have chosen a much larger universe of digital resources, which requires a 
rethinking of terminology, classification schema, and analytical approaches. 

2. Definitions: What are we studying? What do faculty use in their teaching? 

Since the outset, this study has been concerned primarily with unrestricted digital collections that 
support research and teaching in humanities and social sciences. Little did we know when we 
started that this term would engender so much confusion among the diverse groups with which 
we have been conversing. We determined shortly after commencing that we were on the right 
track, but a subtle redefinition of our subject of study was imperative. 

We found, for example, that the term "digital collections" is often used synonymously with 
"digital library collections." This study, in contrast, is primarily focused on unrestricted digital 
resources that may reside in or outside of libraries. Our choice of the term "unrestricted" meant 
we would include resources of digital information that are freely and publicly accessible via the 
World Wide Web, such as mixed media (encoded texts, images, sound files, video, etc.) as well 
as metadata. Such resources proliferate in digital libraries (which have for some years now 
been digitally reformatting materials in special, archival, print, photographic, and other 
resources), museums (which create and distribute digital surrogates for artifacts and works of 
art in their resources), and archives. But they are also created by individual scholars (for 
example, as references to aid their teaching or as supplementary and supportive materials to 
research publications; Read, 2003), and by scholarly initiatives (e.g., the Oxford Text Archive 
and UVA's Institute for Advanced Technology in the Humanities 13 ). They may also include 
resources that reside on a variety of media sites (e.g.. The Neiv York Times ) and in government 
databases (e.g., USGS). 14 As a result, we distinguished such unrestricted digital resources from 
restricted resources of books and journals (e.g., JSTOR, Highwire Press, CIAO at Columbia 
University Press, NAS Press) and those resources that comprise nothing but descriptive 
information for, and pointers to, other information objects (e.g., Melvyl). 15 

Our data gathering reinforced our decision to maintain the broadest possible definition at the 
outset. Faculty, when asked (as opposed to being told what resources are of value), made clear 
that the variety of unrestricted digital resources available to potential users is extensive and 



11 http: / /memory.loc.gov/ammem, http://www.perseus.tufts.edu 
12 http://www.openarchives.org/OAI/openarchivesprotocol.html 
13 http://ota.ahds.ac.uk, http:/ /www. iath.virginia.edu 
14 http://www.nytimes.com, http://www.usgs.gov 

15 http://highwire.stanford.edu, http://www.ciaonet.org, http://www.nas.edu, http://melvyl.cdlib.org 
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diverse; it is not just text and it does not just emanate from libraries or any other single source. 
As to the term unrestricted, it has limited utility. We soon discovered that faculty in research 
universities use a wide variety of licensed digital resources, but consider them to be 
"unrestricted." They make no distinction between, say, RLG Cultural Materials and a digital 
collection such as MOAC. 16 For these faculty, both are accessible; most faculty have no idea 
(until they are told or lose access) that the former is available only because their institution has 
licensed it. Another issue is illustrated by using RLG Cultural Materials and MOAC again as 
examples. The fact that these two digital collections have overlap in their content (and with 
other resources as well) presents a host of complications in understanding user behavior, 
especially around the issue of discovery. In a world where digital objects are represented in 
multiple spaces and places, the user has numerous ways of finding an object and may use 
various pathways to pluck a desired item out of a "collection." 

Therefore, for the purposes of our project, we replaced the ambiguous term "digital collections" 
with "digital resources." Digital resources, for our purposes, encompass those objects that 
employ rich media and span text, images, sound, maps, video, and many other formats. The 
source of these resources is equally broad, and can include formal collections of resources 
developed by well-known institutional entities, such as libraries and museums, as well as those 
developed by individual scholars. Faculty have various types of digital resources from which 
to choose, depending on their given needs. Significant digital resources may be located in the 
deep web, buried and beyond the reach of Google (e.g., databases such as JSTOR, RLG, etc.), or 
behind password-protected course websites. Although we are particularly interested in free 
content, we considered any resource that faculty said they use, regardless of whether it is 
restricted or not. 

B. Understanding use and users: A literature review. 

1. The larger context for understanding technology and the humanities and social 
sciences. 

The overall higher education context of studying humanities/ social science users of digital 
resources is immense and complicated. We outline some major initiatives and thinking here. 
First, there has been an increased interest in understanding users in the digital library realm 
(e.g., Borgman, 2003; Waters, 2004; Grant, 2003; Blanford and Buchanan, 2003; George, 2003; 
Tenopir, 2003; and Marcum and George, 2003). Second, there have been ongoing efforts to 
understand the technology needs of users in the larger H/SS community and the 
undergraduate classroom. The available, and often overlapping, perspectives span professional 
societies, libraries, instructional/educational technology, pedagogical research, and distance 
education. A few of the notable works are discussed below. 

Humanities and technology: In 2000, the Building Blocks project of the National Initiative for a 
Networked Cultural Heritage (NINCH) 17 conducted a survey by field and discipline in the 
humanities. The survey was designed to increase the involvement of humanists in the future 
design of networked computing. NINCH's findings were used to initiate twenty digital 



16 http:// culturalmaterials.rlg.org 

17 http:/ /www.ninch.org 
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projects for humanities resources, aiming at meeting users' expressed needs. Building Blocks 
built on two seminal projects conducted by the J. Paul Getty Trust. 18 

The Digital Initiatives Database Project queried users about electronic resources and generated a 
web-based registry of digital projects that were produced by libraries. 19 This effort appears on 
its face to have stalled, and their list is quite thin when compared to the U.K.'s robust Humbul 
Humanities Hub database of humanities digital resources. 20 Other active U.K. organizations are 
the Resource Discovery Network (RDN), the national Arts and Humanities Data Service, and 
the Digital Resources for the Humanities (DRH) group. 21 

The work of the NINCH and ARL initiatives has clearly influenced, by virtue of common 
sponsors and / or overlapping personnel, a recent effort by the American Council of Learned 
Societies (ACLS) Commission on Cyberinfrastructure for the Humanities and Social Sciences. 
The commission was charged to describe, analyze, and provide recommendations on the 
current state of humanities and social science cyberinfrastructure; its ambitious draft final 
report was published in fall 2005 for comment. The commission report does not devote 
discussions specifically to teaching, however. 

Real and virtual classrooms: There are numerous investigations of how digital resources are 
used in real and virtual classrooms. In our opinion, the most robust work to date in this broad 
area is concentrated on science, technical, and vocational education. Systematic research 
concerned with the humanities is much rarer and, when available, tends to focus on limited case 
studies. 

Much work investigating users of technologies in courses has been accomplished by the 
educational/instructional technology and distance education communities (see, for example, 
publications by ALN Magazine, Syllabus, the Technology Source, Innovate, etc. 22 ) Meta- 
literature reviews are available on the thorny issue of learning outcomes (e.g., Phipps and 
Merisotis, 1999; Waxman et al., 2003). Others are more broadly concerned with economic and 
institutional issues (Fisher and Nygren, 2002, for the A. W. Mellon CEUTT studies; Lorenzo and 
Moore, 2002, for the Sloan Consortium; Twigg, 2003, for the Pew Course Redesigns). In 
addition to the burgeoning literature on distance education in the journals noted above and 



18 The Object, Image, Inquiry: The Art Historian at Work Project (1985-1988) and the Research Agenda for Networked 
Cultural Heritage Papers (1995), http://www.ninch.org/bb/project/history.html 

19 http://www.arl.org/did 

20 http://www.humbul.ac.uk 

21 Links to all of these projects can be found at http://www.ahrc.ac.uk/ictmap. JISC (U.K. Joint Information Systems 
Committee) (2005) has published a strategic plan for 2004-2006, which includes the goals to "ensure ICT is embedded 
within post 16 and higher education, develop eResearch infrastructure and use, and help institutions manage 
investments in ICT." 

22 ALN Magazine. Asynchronous Learning Networks, http://www.aln.org/alnweb/magazine/alnMaga.htm; 
Association for the Advancement of Computing in Education, http://www.aace.org/pubs; The American Journal of 
Distance Education, http://www.ed.psu.edu/acsde/Jour.html; Educom Review, 

http:/ /www. educause.edu/pub/er/erm.html; European Journal of Open and Distance Learning, 
http://kurs.nks.no/eurodl/eurodlen; Journal of Asynchronous Learning Networks, 

http://www.aln.org/alnweb/journal/jaln.htm; TechKnowLogia: International Journal of Technologies for the 
Advancement of Knowledge and Learning, http:/ /www.techknowlogia.org; The Technology Source, 
http://technologysource.org/; Syllabus, http://www.syllabus.com 
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elsewhere, NSF has recently commissioned a literature review about the implications of 
information and communications technologies for distance education. 23 

The U.K.'s Joint Information Systems Committee (JISC) and NSF have just recently funded a 
variety of projects through the Digital Libraries in the Classroom Programme. 24 The charge is to 
investigate the integration of resources into teaching contexts. One project now underway is 
Digital Anthropology Resources for Teaching (DART) at the London School of Economics and 
Columbia University. 25 A report on their progress can be found on their website, although the 
report is primarily focused on technical issues and includes no work on user studies per se 
(Dahlquist et al., 2005). Another JISC/NSF project is The Spoken Word, now being pilot tested 
with instructors and students at Michigan State University and Northwestern University. 26 
Glasgow Caledonian University 27 is digitizing much of the BBC's radio archives into a 
collaborative digital sound archive that faculty in history and political science will be able to 
access, segment, annotate, and wrap into teaching materials. Their website says, "We are 
simultaneously embracing the socio-technological world of the modern learner. The use of 
blogs in teaching and showcasing audio, and the investigation of delivery of audio to mobile 
devices are just two of the ways in which we are responding to the practical needs of our users." 
A perusal of their website indicates evaluation materials related to users is not yet available 
from this project. An evaluation to measure the key impacts of the projects developed under 
the Digital Libraries in the Classroom Programme is being performed separately. 28 

Brogan (2005) in her review of American Literature/ American Studies, describes the profusion 
of resources available to faculty and students in this field, and points to numerous programs 
focused on integrating these resources into the undergraduate teaching. Among the programs 
she references are those developed by individual scholars (Voice of the Shuttle), and larger 
efforts such as the Visible Knowledge Project at Georgetown University and the Center for 
History and New Media at George Mason University. 29 

The American West Project is centered at the California Digital Library and is funded by the 
Hewlett Foundation. 30 This ambitious project has among its goals "to assemble an American 
West virtual collection drawing from the resources of major research institutions." The virtual 
collection "will be assembled and presented with a range of tools supporting extensive 
reconfiguration [and] integration with online learning environments." The development of 
these tools is being informed by assessments conducted with a variety of audiences including 
university librarians and K-12 teachers. 

Improving undergraduate education: There are numerous initiatives that target improving 
undergraduate education in colleges and universities. Many of these are institutional activities. 



23 The National Science Foundation under the Implications of Information Technologies Initiative 
http: / / srsweb.nsf.gov/ it_site / it/infotech.htm. 

24 http://www.jisc.ac.uk/index.cfm?name=programme_dlitc 

25 https:/ / dart.columbia.edu 

26 http://www.jisc.ac.uk/index.cfm?name=project_spoken_word 

27 http:/ /www. gcal.ac.uk 

28 See http://www.jisc.ac.uk/index.cfm7nameHrmding_diglib. As part of the JISC- and NSF-funded Libraries in the 
Classroom Programme, a "Tools Focus Study Final Report" identified and documented the software tools being used 
and developed within the projects so that the sharing of tools across the projects in the program could be facilitated. 

29 http:/ /vos. ucsb.edu, http:/ /crossroads. georgetown.edu/vkp, http:/ /chnm.gmu.edu 

30 http://www.cdlib.org/inside/projects/amwest 
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Some are national in scale. (Links to and descriptions of many of the existing programs can be 
found at the UC General Education Commission website. 31 ) 

One example of an institutional initiative is the Shared Pedagogical Initiative: A Database of 
Electronic Resources for the University of California Community (SPIDER). 32 The project has 
experimented with the integration of resources for teaching writing and research to 
undergraduates. Modeled after a virtual pedagogical seminar, the project has a searchable 
database of modular and peer-reviewed instructional and educational materials for instructors 
and students. Elizabeth Losh, the writing director for the UC Irvine Humanities Core Course 
and SPIDER team member, is consulting on our project. She and her colleagues report that with 
SPIDER, students are doing more research, using more resources at the library, and producing 
better quality writing. Another notable program is Mellon's 2004 Librarian/ Faculty Fellowship 
on Undergraduate Research at UC Berkeley. 33 It aspires "to create a program that encourages 
and facilitates faculty collaboration with the library and other partners to build undergraduate 
knowledge of information resources; enhance student research and information competencies; 
connect faculty research more effectively with classroom teaching; and provide extended 
opportunities for faculty to mentor creative student discovery and research both within and 
beyond the classroom." 

An example of a national initiative is the Reinvention Center at Stony Brook University, which 
focuses on undergraduate education at research universities. 34 Among its work is the 
sponsorship of various conferences including those on the integration of research and 
education. It has a new initiative to sponsor forums focusing on undergraduate scholarship in 
the humanities. 

The Carnegie Foundation for the Advancement of Teaching has a number of ongoing projects 
relevant to the improvement of undergraduate education. These include Initiatives in Liberal 
Education (ILE), the Integrative Learning Project, the Carnegie Academy for the Scholarship of 
Teaching and Learning (CASTL), and the Knowledge Media Laboratory (KML), which 
"develops tools and resources to exchange information, share knowledge and produce 
innovations that can transform teaching and learning at many levels." 35 The Association of 
American Colleges and Universities' (AAC&U) "Greater Expectations" report (Ramaley et al., 
2004) evaluates the state of undergraduate general education. AAC&U has numerous other 
initiatives in general and liberal arts education. 36 

The AAU report "Reinvigorating the Humanities" (2004) makes recommendations for 
improving undergraduate teaching and also describes a variety of outreach innovations taking 
place between universities and K-12. 

The National Institute for Technology and Liberal Education (NITLE) "serves as a catalyst for 
innovation and collaboration for national liberal arts colleges as they seek to make effective use 
of technology for teaching, learning, scholarship, and information management." 37 A major goal 



31 http://cshe.berkeley.edu/gec 

32 http:/ / eee.uci.edu/ programs/ spider 

33 http://library.berkeley.edu/Staff/MellonProject 

34 http:/ /www. sunysb.edu/Reinventioncenter 

35 Links to these initiatives can be found at: http://www.carnegiefoundation.org/ourwork. 

36 http:/ /www.aacu.org/issues/generaleducation 

37 http:/ /www.nitle.org 
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is to facilitate the development of dynamic curricula that are responsive to innovations in 
information technology. 

The Associated Colleges of the Midwest (ACM) received a grant from the Mellon Foundation to 
better understand student experiences with their liberal arts education (freshman year 
experiences, capstone experiences, etc.). This project mainly involves liberal arts colleges and 
the University of Chicago, and is described in "Engaging Today's Students with the Liberal 
Arts" (2005). 

Information literacy projects: There is significant attention focused on "information literacy" 
projects, in which students and instructors are the primary audience. They are too numerous to 
review here, and are often closely linked to the above Undergraduate Education Improvement 
Initiatives. The Association of College and Research Libraries has developed Information 
Literacy Competency Standards for Higher Education. 38 The A.W. Mellon Foundation has 
funded a multi-year collaborative project between Connecticut College, Wesleyan University, 
and Trinity College for librarians and faculty to develop and host information literacy 
workshops for other faculty and administrators, and to build a collaborative database for 
sharing information literacy learning modules and teaching resources. 39 The Committee on 
Information Technology Literacy (1999) published a major National Academy of Sciences report 
on the effective use of information technology among college students. And more recently, 
Tobin (2004) has presented a review of "Best Practices for Online Information-Literacy 
Courses." 

Developing effective tools for instructional use of digital resources: The integration of 
diverse digital resources into teaching practice, especially the integration of library resources 
with learning management systems, is seen as a pressing problem (McLean and Lynch, 2003; 
Flecker and McLean, 2004). We are struck by the many parallel tool development initiatives 
taking place throughout higher education, both domestically and internationally. We suspect 
most users are similarly confounded by the many options available or under development. 

Minielli and Lerris (2005) provide a description and literature review of the use of course 
management systems (CMS) in higher education. There is a significant effort toward the 
development of open-source software tools to support teaching (e.g.. Open Knowledge 
Initiative at MIT 40 ). The Sakai project is a consortium of universities developing open-source, 
standards-based, and extensible collaborative learning environments. 41 Rather than focusing on 
open courseware applications, Sakai is designed to compete with for-profit course management 
systems, and includes a suite of CMS tools. The open online educational resource space 
includes MIT OCW software and content development efforts; eduCommons at Utah State 
University, Rhaptos from Rice University's Connexions project, and Melete, which is part of the 
ETUDES project at the Loothill De Anza Community College District. 42 

The LionShare P2P project, based at Penn State University, is an effort to facilitate legitimate file 
sharing using Peer to Peer (P2P) technology for the easy exchange of image collections, video 



38 http://www.ala.org/acrl/ilcomstan.html 

39 http :/ / camel2.conncoll.edu /is/ infolit 

40 http:/ /www. okiproject.org 

41 http:/ /www. sakaiproject.org 

42 http:/ / cosl.usu.edu/projects/ educommons, http:/ /rhaptos.org, 
http: / / foothillglobalaccess.org/etudes2/melete.htm 
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archives, large data collections, and other types of academic information. 43 LionShare is open 
source and should ultimately provide users with resources for organizing, storing, and 
retrieving digital files. 

Brogan (2005) reviews tools specifically for humanities scholars, including NITLE Semantic 
Engine, designed to access and organize unstructured digital text; the NORA project, which is 
developing software for discovery, visualization, and exploration across large full-text 
collections; and the DLF Aquifer project, which is developing a testbed of library tools and 
services for aggregation and distribution of content. 44 

The 2005 Summit on Digital Tools for the Humanities brought together scholars to "assess the 
state of development of digital tools for humanities research, as well as the effectiveness of the 
supporting and integrating cyberinfrastructure." 45 Funded by NSF and hosted at the Institute 
for Advanced Technology in the Humanities (IATH), the summit will result in a final report. 

The management, preservation, and dissemination of the ever-expanding digital material that 
higher education institutions create in the research and teaching realms is a current challenge. 
According to Lynch and Lippincott (2005) and Westrienen and Lynch (2005), a large number of 
institutions they surveyed in the U.S. and abroad have developed or will develop institutional 
repositories (IR). 46 They note that IRs are becoming well established as campus infrastructure 
components. As broadly construed, and apparently as understood by those surveyed in the 
above studies, an IR should house not just e-prints, but datasets, video, learning objects, 
software, theses, and other materials. 

In an article on the emerging need for supporting personal collections, Beagrie (2005) points to 
the variety of commercial and non-commercial products emerging that support the increasing 
need of people to capture and store personal digital information, including emails, documents, 
articles, portfolios, and digital images, video and audio. Examples of software and services 
include MyLifeBits, Lifeblog, Data Deposit Box, and Ourmedia. 47 As he notes, processing, 
storage, and software tools available to individuals are increasing in power, volume, and ease of 
use, and will provide new ways in which individuals can create, manage and disseminate a 
diverse range of media types. 48 

Given this brief background, we focus below on those robust studies that specifically attempt to 
understand users of digital resources. 



43 http:/ /lionshare.its.psu.edu/main 

44 http://www.nitle.org/tools/semantic.htm, http://nora.lis.uiuc.edu, http: //www. diglib.org/ aquifer 

45 http : / / www .iath. Virginia . edu/ dtsummit 

46 DSpace, which was developed at MIT, is perhaps the best-known example. The Portico project 

(http:/ /www.portico.org) provides a new model for a sustainable electronic-archiving. Its mission is to preserve 
scholarly literature published in electronic form and to ensure that these materials remain accessible to future 
scholars, researchers, and students. 

47 http://research.microsoft.com/barc/mediapresence/MyLifeBits.aspx, http://www.nokia.com/lifeblog, 
http://www.datadepositbox.com, http://www.ourmedia.org 

48 See also Firefox Scholar, a plug-in to the Firefox browser, under development at GMU 

(http:/ /chnm.gmu.edu/tools/firefoxscholar). It promises to organize citations on the desktop by "automatically 
[capturing] author, title, all that info that scholars want to save." (Young, 2005). 
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2. Specific research on digital resource users 

Quite simply, existing user research is as diverse as the resources available for study and the 
motivations for understanding users. As a result, there is no single, uniform approach that can 
be gleaned from these available studies. 49 For example, and described in more detail below, 
there is a relative hodgepodge of excellent studies that are germane to the improvement of 
targeted projects (e.g., Perseus, MO AC, Alexandria Digital Library 50 , MIT OCW, Carnegie 
Mellon's Online Learning Initiative) or reflect broad surveys of librarians and library users in 
academic settings. Electronic journal and library use studies may be the most abundant. In 
addition to studies such as Troll Covey (2002), which specifically addressed the types of 
methods for use and usability of digital library resource collections, there are library studies 
that, for example, have looked at data mining (e.g., Mento and Rapple, 2003). Meta-research 
projects, which compile and analyze findings from multiple user studies, offer valuable insights 
but are limited in number and scope, and each study has its own limitations in the context of 
our project. 

We have organized existing user studies into four primary areas: (1) electronic resource/ digital 
library use studies, (2) cultural heritage research, (3) evaluations of specific sites, and (4) image 
services. We also identify a fifth emerging area, complex media environments, for which, to our 
knowledge, robust user studies per se are not yet available. These rich media environments 
include N-way video. Global Information Systems (GIS), virtual reality, simulations, and games. 
As mentioned above, there is a burgeoning literature on educational technology assessment and 
evaluation in general. There is also a smaller literature focused on the question of cost 
effectiveness of educational technologies in various academic environments. Neither of these 
areas will be covered in detail here. 

Electronic resource and digital library use studies: Perhaps the most exhaustive meta-research 
project is a recent report from the Council on Library and Information Resources (CLIR) 
(Tenopir, 2003). The research summarizes findings about the use and preferences for print and 
electronic services in academic libraries, drawing from 200 different studies published between 
1995 and 2003. The report concludes that, overall, experts in different disciplines have different 
usage patterns of digital resources, and that students and faculty alike are more likely to adopt 
electronic resources if they are convenient, relevant, and save time. This valuable study 
however, is too "library-centric" for our purposes and does not include research about the use 
of text or non-text resources developed outside of library contexts. 

Ithaka 51 and the Andrew W. Mellon Foundation recently completed a second survey of faculty 
at four-year U.S. higher education institutions to learn about use of electronic resources. More 
than 7,000 faculty members responded to the survey. Among the published findings is the fact 
that even though faculty rely on electronic resources, they still see barriers to use. Differences 



49 Khoo and Ribes (2005) noted the "diverse range of research methods" used by participants in a workshop focused 
on creating a dialogue among researchers involved in qualitative analysis of digital library users. The Cultural 
Content Forum (http:/ /www. culturalcontentforum.org) made an initial foray into solving this problem for cultural- 
heritage institutions (Alice Grant Consulting, 2003). 

50 http:/ /www.alexandria.ucsb.edu 

51 http://www.ithaka.org 
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among disciplines on various measures of use and satisfaction have also emerged (Kiernan, 
2004; Schonfeld and Guthrie, 2004 52 ). 

The Institute of Museum and Library Services (IMLS) 53 has funded a major Online Computer 
Library Center (OCLC) study of electronic research titled, "Sense-Making the Information 
Confluence." The project's goals are to understand the hows and whys of electronic resource 
use (Dervin et al., 2004). 

Brockman et al. (2001) developed user-based criteria for guiding digital library development by 
studying what scholars do in research and writing. They conceptualize the type of information 
environment that would best support humanists' activities and make recommendations on how 
information environments can be developed in such a way as to be responsive to the context of 
scholarly work. In the preface, Dan Greenstein emphasizes that the study provides a variety of 
lessons, including developing collections that "support specific research aims and thus are 
formed in close consultation with the scholars who share these aims. " 

The EPIC Online Use and Costs Evaluation Project at Columbia University found that a wide 
variety of electronic resources is used regularly by faculty and students for research, teaching, 
coursework, communicating with colleagues, or just looking up general information related to 
their academic work (Electronic Publishing Initiative at Columbia, 2005). When teaching, 
"faculty often 'surf' to see what information is available both in online databases and in other 
Internet sources to help demonstrate ideas, give a current context to the lecture material, or find 
a scientific database that can be used in class." Electronic resources are seen as providing 
increased convenience and increased access to information. Disadvantages included sorting 
through good and bad sources, and just having too much information available to sort through. 

Foster and Gibbons (2005) focused on understanding "the apparent misalignment between the 
benefits and services of the DSpace institutional repository (IR) with the actual needs and 
desires of faculty." Their work entailed interviews and observing how University of Rochester 
faculty members interact with digital tools and how they organize work in their virtual and 
physical workspaces. Their findings resulted in a redesign and better alignment of the 
repository with how faculty work. The findings also resulted in a rethinking of how to explain 
and promote the local IR. 

The "E-journal User (EJUST) Survey" at Stanford (Keller, 2002) was not focused on applications 
in the teaching contexts. Among its findings, however, was that e-journals improve the 
efficiency of scientific scholarship. Electronic search engines and online access to abstracts and 
full-text articles speed up the process of searching and retrieving relevant scholarly content. 
Another conclusion was that e-journals facilitate new forms of scholarly practice through new 
relationships to information, knowledge, and peers. 

Cultural heritage research: Museums and other cultural heritage institutions, which have a 
unique and successful history of balancing curatorial demands with public education, are a 
burgeoning area of user research. These studies often overlap with the digital library sector. As 
an example, there are a number of relevant papers in the proceedings from the Fifth Annual 
Conference on Libraries and Museums in the Digital World (2004). Hamma (2004), for instance. 



52 JSTOR, a History (Schonfeld, 2003), presents a detailed analysis of JSTOR, a success story that can be measured by 
its widespread use. 

53 http://www.imls.gov 
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describes user research at the Getty Museum that resulted in a redesign and rethinking of that 
institution's site to meet the needs of three different types of user profiles. 

The Cultural Content Forum, 54 based in the U.K., commissioned research to identify, analyze, 
and disseminate material related to the evaluation of digital cultural heritage resources (Alice 
Grant Consulting, 2003). The 2003 report attempted, through a survey of cultural heritage 
institutions, to analyze multiple user studies. Its primary goal was to seek an intersection of 
metrics used across common sites. It also developed user-profile characteristics that could help 
generate a standardized profiling approach. The work promised further exploration of two 
areas: (1) the identification and definition of metrics and measurements used in evaluation 
projects, and (2) the identification and definition of a range of user profiles for use in evaluation 
work and for establishing methodologies to facilitate comparison across projects and domains. 
We have not had luck in following up with this group, however, and assume the project is no 
longer active. 

Zorich (2002) conducted a survey of North American-based digital cultural heritage initiatives 
(DCHIs). The purpose of the survey was to identify the scope, financing, organizational 
structure, and sustainability of DCHIs. With reference to users, she notes that many DCHIs 
began their activities with no knowledge of levels of user interest and needs. "Many projects 
found their usage was much less than anticipated. Even now, most DCHIs feel that no one 
really understands what users want, despite a recent increase in studies of user needs." 

A nationwide 2006 survey by the Institute of Museum and Library Studies of 947 libraries, 
museums, and archives concluded, "Three-quarters or more of institutions in all groups do not 
conduct assessments of user or visitor needs. ... Almost one-fourth of state library 
administrative agencies do assessments, which is the highest level among all the groups." (The 
Institute of Museum and Library Studies, 2006) 

In the process of creating a guide to business planning aimed at cultural heritage institutions, 
Bishoff and Allen (2004) conducted a survey of thirteen institutions. They concluded that these 
institutions had varying levels of experience with market research, needs assessment, and 
outcomes assessment. They also concluded that, when compared to libraries, museums must 
have a good understanding of their markets to develop strategies to maintain or increase gate 
receipts, an essential source of revenue. Museums therefore tend to break down their "visitors" 
into several categories. 

Site-specific user studies: Marchionini (2000; Perseus), Gilliland-Swetland (1998; MO AC), 
Borgman et al. (2001; Alexandria Digital Library), Hill et al. (1997; Alexandria Digital Library), 
and Carson (2004; MIT OpenCourseWare) are excellent examples of rigorous "product testing" 
user studies that employ a multiple-data collection strategy or "portfolio" approach. All of 
these studies, to some degree, have employed a combination of surveys, discussion groups, 
follow-up interviews, and transaction log analysis (TLA) to get a broad look at site-specific user 
and developer behavior. This technique is apparently relatively new to the library world, 
although it has been used on educational sites for some time (e.g., Harley et al., 2002 and other 
CEUTT studies 55 ). Carnegie Mellon University's Online Learning Initiative (OLI) is conducting 
several studies that combine an understanding of student learning with the implementation of 



54 http:/ /www. culturalcontentforum.org 

55 http:/ /www. ceutt.org 
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stand-alone online courses. 56 They are using a variety of methods including student 
demographic information as well as perception and attitudes (through entry, exit, and 
embedded surveys), instructor information (through casebooks, surveys, and workshop 
feedback), usability studies, effectiveness studies, evaluation of student achievement, and 
spontaneous unsolicited feedback. 57 

Triangulation of multiple methods in these studies contributed to the ongoing redesign of the 
sites for multiple audiences. They also set an emerging standard for methods necessary to an 
understanding of users, and to evaluating digital resources in complex educational 
environments. In all cases, a major goal has been the integration of the sites into 
teaching/learning environments. These studies, taken together, may point the way toward 
assessing the value of user studies for the user of specific resources. 

Image service studies: Art history, a field that has depended heavily on slide libraries for 
teaching, is a fertile area for the integration of digital imaging. Small case studies suggest 
significant pedagogical and cost-savings potential for art history faculty (e.g., Burnett et al., 
2002 ). 

Penn State's Visual Image User Study (Pisciotta et al, 2002) is an excellent internal study and 
needs assessment of image services that, in addition to assessing that campus' climate and 
needs, pointed to the limitations of current software, as well as the importance of faculty 
personal digital collections and resources. 

The Research Libraries Group (RLG) 58 is currently involved in various activities to make their 
cultural materials more amenable to classroom use. RLG has conducted interviews with faculty 
to probe how they use digital images now, the barriers they encounter, and what they would 
like to see happen in an ideal world. Among their findings are that faculty use their own 
materials, rely on PowerPoint for presentation, and are not dependent on aggregated digital 
collections such as RLG (and may not know about them). The importance of the Google search 
model is clear, and in a perfect world, the idea of searching across all licensed resources and the 
web at the same time found many proponents (Waibel, 2004 59 ). 

The California Digital Library (CDL), 60 one of our partners and with whom we have consulted, 
conducted an internal assessment of its image demonstrator service, which uses Luna Insight 
technology (Farley, 2004). The results of their internal evaluation mirror those of Penn State, 
RLG, and our own findings. Faculty use of personal digital resources is important, and effective 
tools to manage these resources and reuse them in new contexts need to be developed. 

ARTstor is currently conducting an ongoing internal evaluation of their own collection and 
services. Like other sites, ARTstor has found that ease of use and reusability are primary 
concerns among users (OER Meeting, 2005). In addition to needs assessments that help the 
organization make critical decisions about resource investment, a formal survey found that 
while there appeared to be a greater overall dependence upon digital resources, disciplinary 
differences were apparent in attitudes of faculty toward and use of digital resources (Shonfeld 



56 http://www.cmu.edu/oli/research/index.html 

57 Personal communication with Candace Thille, Director of OLI. 

58 http://www.rlg.org 

59 Also personal conversation with RLG staff members Merrilee Proffitt and Gunter Waibel. 

60 http://www.cdlib.org 
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and Guthrie, 2004). These early findings suggest that the development of tools and services may 
need to be customized for specific disciplinary needs. 61 

3. Complex new media (interactive video, GIS, 3D applications, games, social 
software) 

It is generally agreed that humanists and social scientists will depend on complex media tools 
to realize the full potential of digital resources in their teaching and research (ACLS 
Commission, 2005; HASTAC, 2004 62 ), though this has not yet been fully realized (Ayers, 2003). 
The potential emergence of robust applications employing GIS, N-way video 63 , and virtual 
reality may finally offer tangible opportunities for those in the humanities and social sciences 
who want to integrate such technology into their scholarly and pedagogical practice. The 
embryonic and fragile nature of some of these technologies in real teaching and learning 
contexts, however, has kept them out of the hands of anyone but those with the most funding 
and guts, so there are few significant data on their use in undergraduate educational contexts. 

Ayers, who has been instrumental in developing innovative approaches to digital scholarship, 
has been a champion of rethinking how the online environment can go beyond the simple 
contextualization of knowledge to a space for nonlinear teaching and learning. Specifically in 
his work with the Institute for Advanced Technology in the Humanities (IATH) and the Valley 
of the Shadow project 64 , he has demonstrated how the basic IT infrastructure can be harnessed 
to support unique ways of content development and creative reuse (Ayers 2003, 2004). 

More robust applications of virtual reality are being developed and utilized, such as the 
Cultural Virtual Reality Lab, which recreates the Roman forums, and MIDA (Mellon 
International Dunhuang Archives), which allows scholars to view Chinese cave paintings in 3D 
using virtual reality technology (cited in ACLS Commission, 2005). And, we are seeing more 
interest in applications which rely on spatial data, such as ECAI (Electronic Cultural Atlas 
Initiative). 65 

Since we embarked on this report, many discussions have emerged around social-media 
applications in support of scholarship (if not always H/SS focused). Various genres of "social 
computing/ software" in particular have garnered an exceptional amount of attention 66 . Social 
software and social computing as they are broadly discussed in the popular press cover a wide 
variety of applications, activities, and projects, including blogs, wikis, audio and video remix, 
podcasting, MySpace, Facebook, Wikipedia, del.icio.us, and Flickr, to name just a few. 67 Much 
of the enthusiasm is generated by interest in social-network theory, and the ability of users to 



61 ARTstor is undertaking further qualitative investigations to better understand the kinds of organizational 
structures and local support necessary (Personal conversation with Ithaka staff member Roger Shonfeld, November 
28, 2005). 

62 The vision statement of HASTAC (Humanities, Arts, Science, Technology Advanced Collaboratory), which was 
founded by a number of national humanities centers, supercomputing centers and other entities, lays out a vision for 
collaborative possibilities in e-humanities (http://www.hastac.org). 

63 Kaufmann (2005) describes the importance of video and television for education and envisions a future of open 
production. 

64 http://valley.vcdh.virginia.edu 

65 http:/ /www. cvrlab.org, http://www.artstor.org/info/collections/mida.jsp, http://ecai.org 

66 See, for example, http://www.web2con.com 

67 http://www.myspace.com, http://www.facebook.com, http://www.wikipedia.com, http://del.icio.us, 
http://www.flickr.com 
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create and share social tags (keywords) on any and all web content without regard to rigid 
centralized categorical constraints (Flickr and del.icio.us being the most notable examples). 
Examples of podcasting of lectures and other audio, and the use of blogs and wikis in writing 
courses can be found in any publication focusing on technology in higher education. 

Wikipedia, as described on its website, is a "multi-lingual Web-based free-content encyclopedia. 
It is written collaboratively by volunteers, allowing articles to be added or changed by anyone 
with an Internet connection." It is of particular note because its distributed authoring model 
has been hailed by enthusiasts as the model for creating everything from electronic college 
textbooks to scholarly publications. The degree to which such a model can ensure high quality 
and avoid misinformation is very much under debate, however (Siegenthaler, 2005). 
Experimentation with these types of technologies in pedagogical contexts abounds and will 
surely increase. 

There has also been significant attention focused on how games and simulations might be used 
for educational purposes (including MMORPGs, Massively Multiplayer Online Role-Playing 
Games). Proponents argue that games have the potential to teach higher order thinking skills as 
well as encourage adaptation to continuously changing environments (e.g., Kelly, 2005, and 
NESTA Futurelab, 2005). Kelly specifically suggests that while games cannot replace traditional 
forms of teaching or teachers, they can offer similar functions as one-on-one tutors without the 
drain on teacher time. John Seely Brown (2005) further argues that in specific interest 
communities, including online role-playing games, the range of "users" from novices to experts 
and the interactions that occur enable a kind of peer modeling or informal apprenticing that 
teaches not only actual skills but situational knowledge as well: the culture, sensibilities, and 
aesthetics — or epistemological framework — that make up that profession or role. 68 For 
educational games to be widely used in teaching, Kelly (2005) argues that significant funding 
for research and development of tools, standards, infrastructure for data collection, and 
evaluation of effectiveness is necessary, especially in the area of "serious games" or games that 
incorporate real educational skills in order for the player to advance. Gee (2005), however, 
suggests that many commercial games are already based on good theories of learning. 

The new generation of students that higher education can expect to welcome is variously 
known as the Net Generation, ikids, and the "always on" generation. They depend upon and 
expect convenience and easy access to information, and often find linear learning unfamiliar 
and difficult (Ayers, 2003, Oblinger and Oblinger, 2005). A UC Berkeley and USC Annenberg 
Center project funded by the MacArthur Foundation has just begun an ethnographic study to 
investigate the broad outlines of how kids are using various types of new media in informal 
learning contexts 69 

It is expected that these students will be characterized by their facility with mobile devices and 
their willingness and interest in creating digital content through blogs, social tagging, remixing 
audio, etc. Many expect the emergence and evolution of new mobile devices such as phones 
and iPods (Lenhart and Madden, 2005; Ito et al, 2005), which allow access to unlimited 
information in a pocket and enable communication via text, video and audio, to have profound 
effects on learning. The recent ECAR report (Kvavik and Caruso, 2005), however, suggests that 
current college students may have fewer demands for technology on the higher education 



68 Also see http://www.johnseelybrown.com/learning_in_digital_age-aspen.pdf 

69 "Kids' Informal Learning with Digital Media."; http://groups.sims.berkeley.edu/digitalyouth; 
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landscape than previously expected. It is unclear whether the student desire for "moderate" 
technology in learning may change as the younger generation enters higher education. 

In summary, we can continue to say with confidence that there has been no coordinated 
conversation about the "why" and "how" of user studies that could apply across the many 
types of digital resources and their sources. This is in large part related to (1) the relatively 
recent availability of multiple, well-developed digital resources, (2) the great diversity of digital 
resource types that has emerged, (3) the significant costs of well-designed user research, (4) the 
multitude of user types, potential educational contexts, and motivations for studying use, and 
(5) the evolving (moving target) nature of the digital resources themselves. The latter has 
required that evaluators focus both on "product testing" paradigms and on research that 
analyzes the "interactions of complex phenomena" (Marchionini, 2000). 

C. Developing a typology — but from whose perspective? 

Any attempt at describing the universe of faculty use of digital resources demands some 
common vocabulary. But there are challenges to the construction of such a vocabulary, not least 
of which is that digital resources of all kinds are proliferating in many different environments 
and are created by many different kinds of developers. The confusion can be seen in three 
areas: 

• Defining any collection of digital resources is complicated by whose perspective you 
take. 

• Different users may view and value the digital resources available to them differently 
from one another and differently from those who create and manage digital resources. 

• The varied proprietors of digital resources have different views and roles, and may 
value resources and collections differently. 

Based on discussion with a Site Owners Advisory Group, our faculty discussion groups, and 
interviews of and a meeting with digital resource providers, it is clear that the array of available 
digital resources may represent different things to different types of owners and users. Clearly, 
the perspective of users and owners is quite different both among them and between them. For 
example, what we might describe and label as a collection has little meaning to the typical user. 
Many users are not particularly interested in the word "collection" unless it represents a tightly 
focused collection around a specific topic or topics (e.g., the Jack London Collection 70 ). 

Therefore, the word collection, as used in the digital library world, may be problematic both in 
understanding users and in constructing a map of the universe of "stuff" users want to access. 

In our project's context, which has been to ascertain what people use and not to tell them what 
particular resources are valuable, the issue of how to define digital collection frequently arose. 

We concluded that, while collection owners may indeed care about collections, individual users 
probably do not. Users often have a different level of granularity that categorizes their 
definition (e.g., whether they can find on the web a format, a photo, a picture, or a passage). For 
users, the information needed is frequently a small slice of a larger digital collection. We 
discuss the issue of different perspectives in some detail below. 



70 http:// sunsite.berkeley.edu/London 
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1. The user perspective 

Faculty (and others) have various types of digital resources from which to choose for teaching, 
depending on their needs. As mentioned previously, the resources faculty draw from include 
their own resources, department resources, local library resource collections, licensed resource 
collections, and of course, what is available through Google and similar search engines. 

To complicate matters futher, there is no single type of academic user, nor do users see the 
world in the categories constructed by researchers and librarians. 71 For example, users simply 
want the right object at the right time. That object might be mined from a traditional collection or 
it might be found in any number of spaces — not infrequently, as the result of a Google- type 
search. The types of users who access these resources are also diverse. They range from K-12 
teachers and students to research scholars to undergraduate educators in vastly different types 
of institutions to the general public. Furthermore, these categories of users often comprise 
diverse individuals with varying and idiosyncratic needs, perceptions, and ways of finding and 
utilizing digital resources. 

2. The site "owner" perspective 

One attendee at the Site Owners' Advisory Meeting mentioned above aptly pointed out that the 
terms "owners" or "collection developers and owners" recurred as though they described a 
single group. 72 In reality, though, there is often a more complicated set of roles under the 
designation "owner," and the individuals in those roles ordinarily have different interests, 
values, and, especially, different levels of access to traces of user behavior. He suggested the 
following distinction between these roles: 

• Aggregators, who select which digital resources are to be available in what 
combinations, and try to bring them to the attention of users (e.g., someone at Berkeley 
choosing which resources should be combined in a meta-search service, or someone 
assembling a portal, or RLG in their role of assembling the Cultural Materials service 
and promoting it, or Google) 

• Developers of tools, who shape user interactions, export mechanisms, and access paths. 
This role includes both searching within stand-alone digital resources and making 
digital resources available for federation or crawling or other forms of discovery 

(e.g., developers at Luna in their role of developing Insight, or those developers 
involved in ARTstor in their role of developing a browser-based client and Offline 
Image Viewer, or Blackboard, or RLG in their role of defining and developing 
functionality and export for Cultural Materials, or Endeavor) 73 

• Content creators/ owners, who conceive, assemble, describe, and digitize content 
(e.g., the site development team comprising a programmer at one institution and a 
content developer at another, who both contribute to site development). 



71 Results from our faculty survey and discussions. 

72 Arnold Arcolio of RLG contributed the ideas about multi-faceted owner roles at the May 17th, 2004, Site Owner 
Advisory Meeting. 

73 http:/ /luna-imaging.com/insight, http:/ /www. artstor.org/info/news/oiv_2point5.jsp, 
http://www.blackboard.com, http://www.endinfosys.com 
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3. Typology development 

To describe the complex and diverse world of digital resources, we began by constructing a 
simple typology in spring 2004. We leveraged findings from our fall 2003 faculty discussion 
group sessions and drew from existing research (Brogan, 2003) to develop our initial 
framework. We began with one primary dimension based on types of resources that were 
mentioned by faculty and from our background research of existing resources (e.g., ancient 
manuscripts, image databases, online journals, etc.). The resulting classification is shown in 
Table 1.1, below. We knew before we started that there is often significant overlap among 
potential types. For example, what features distinguish a "digital archive" such as MOAC from 
a broader "non-library /non-museum" resource such as Perseus? How does a set of archived 
course videos differ from the online course materials being mounted in MIT OCW or the 
learning objects of MERLOT? 



Table 1.1: Typology of digital resource landscape 



Types of resources 


Sources of resources 


Images or visual materials (drawings, photographs. 


Search engines/ directories (e.g., Google, Yahoo) 


art, posters, etc.) 


Personal collection of digital materials 


Maps 


Public (free) online image databases 


Simulations or animations 


Commercial image databases (e.g., Saskia, 


Digital film or video 


AMICO 74 ) 


Audio materials (speeches, interviews, music, oral 


Campus image databases from one's own 


histories, etc.) 


institution (e.g., departmental digital slide 


Digital facsimiles of ancient or historical 


library) 


manuscripts 


"Portals" that provide links or URLs relevant to 


Online or digitized documents (including 


particular disciplinary topics 


translations) 


Online exhibits (e.g., from museums) 


Government documents 


Library resource collections (i.e., digital) 


Data archives (numeric databases; e.g., census 


Online journals (e.g., JSTOR) 


data) 


Media sites (e.g., NPR, New York Times, CNN, 


News or other media sources and archives 


PBS) 


Online reference resources (e.g., dictionaries) 
"Portals" that provide links or URLs relevant to 
particular disciplinary topics 
Personal online diaries (e.g., weblogs) 

Online class discussions (including archived 
discussions) 

Curricular materials and websites that are created 
by other faculty or other institutions (e.g., MIT 
OpenCourseWare, World Lecture Hall, Merlot) 
Digital readers or coursepacks 
Interactive Media (e.g., VR, games) 

Tutorials 

Other types of resources 


Other sources of digital resource 



Further complicating a typology is the need to add functional dimensions to that which is 
purely descriptive. In Table 1.2, below, we added both (1) a dimension of "findability," or the 
sources of resources used to discover and locate digital resources (e.g., search engines, portals, 
online exhibits, etc.), and (2) a dimension of digital resource characteristics, or the underpinnings 
of digital resource origination, purpose, audience, and administration (e.g., the original 



74 http://www.saskia.com, http://www.amico.org 
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intended audience, primary or secondary integration into the classroom, university or public 
institution based, foundation funded, etc.). 

Table 1.2: Digital resource characteristics/digital resource provider interview focus 



Focus and goal of site 

Scope (e.g., degree of specificity — Dime Novel 
Collection 75 vs. broad digital resource collections 
which may be a conglomeration of many 
individual sites, like Sunsite 76 ) 

Diversity (how many subsites make up the main 
site?) 

Overall digital resource type (anthology, online 
exhibit, referratory, etc.) 

Overall size (small/large, number of pages, etc.) 

Intended audience (students, instructors, scholars, 
etc.) 

Media content (what types of media are included — 
images, maps, games, tutorials?) 

Digital formats (form of representation, e.g., jpegs, 
tiffs, etc.) 

Content evaluation (what percentage of content is 
evaluated, e.g., refereed?) 

User access (free and unrestricted vs. password 
protected, etc.) 

Integration intention (is the site intended to be 
integrated with other learning resources, CMS, 
bibliographies, etc.?) 

Reuse (to what extent can the content actually be 
reused and by whom; e.g., wrapped in lesson 
plan, jpeg downloaded, etc.?) 

Technical compliance (e.g., Z39.50) 

Metadata (LC subject headings, Dublin Core, none, 
etc.) 

Searchability (Google licensing, federated, 
homegrown search feature, browsing, none) 

Linking (external vs. internal vs. none) 



Management and administration 

Organizational affiliation of resource (public, 
private, consortium, etc.) 

Ownership (single owner, group, museum staff, 
multi-institutional) 

Staffing and roles (how many and in what roles, 
librarians, designers, scholars, etc.) 

Dissemination/Marketing (personal, institutional, 
registered) 

How long the has the site been up (persistence) 

Preservation and maintenance (how often the site is 
updated, expected lifespan) 

Funding (start up and maintenance costs) 

Funding sources for development and 
sustainability (foundations, institutional support, 
income, none) 



Understanding use 

Monitoring usage (what usage data are collected, 
e.g., TLA, anecdotal, surveys, etc.) 

How frequently usage is monitored (monthly, 
yearly, etc.) 

Percentage of budget, per year, spent monitoring 
use 

Percentage of staff, per year, dedicated to 
monitoring use 

Applying usage data (how are usage data applied 
to the operation?) 

Unknown information about use (e.g., what do you 
want to know that you haven't been able to 
collect, or why is understanding use important?) 



4. Usefulness and limitations of a typology approach 

Since our initial typology was based on actual discussion group data, it provides a useful map 
for describing digital resources from a faculty perspective. Together, using all three dimensions 
(type, source, and characteristics), the typology conceivably functions as a theoretical matrix for 
describing the variety and complexity of unrestricted digital resources in humanities and social 
science undergraduate education. But, as with most typologies, which are simplifications of 
complex phenomena, our typology has its limitations. 

In May, 2004, we asked our Site Owner Advisory Group for critiques and revisions of our 
typology. While the group considered our typology a realistic start to describing the digital 



75 http : / / sulair. Stanford . edu /depts/dp / pennies 

76 http:// sunsite.berkeley.edu 
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resource landscape, they acknowledged existing gaps and limitations to our approach. They 
stated a need for adding resource characteristics to our typology, including: (1) a category for 
users' motivation for resource use, (2) a category for different forms of representation in 
resources (e.g., jpeg, tiff, and so on), (3) a category covering implications of reuse (licensed vs. 
unlicensed sites), and (4) a category covering the abilities of users to repurpose content from a 
digital resource collection for their own uses. Arnold Arcolio of RLG suggested that we modify 
and simplify our typology approach altogether, so that resource characteristics are gathered 
around "centers of value," which appear in Table 1.3. These centers of value can function as 
broad yet significant guiding principles, with considerable strengths in describing all kinds of 
unrestricted resources (whether the resource is MOAC, Sunsite 77 , MIT OCW, or a homegrown 
site put together by an anthropology professor for classroom use). 

Table 1.3: Digital resources and centers of value 

• Content coverage (chronological, geographic, thematic, disciplinary, type of "original" — 
manuscripts, coins, maps, games) 

• Form of representation (i.e., availability of digital formats and portability, e.g., jpeg, tiff, sid; 
proprietary or open, level of metadata: structured, standard, rich or thin; wrapper issues, 
e.g., HTML, XML, METS) 

• Authority (e.g., source, maintenance, institutional affiliation) 

• Permitted uses and digital rights of reuse 

• Persistence (e.g., how long is the resource up, how often does updating occur?) 

• Exposure for discovery (e.g., searching paths, browsing, availability for federated search, 

availability for Google crawling) 



At the beginning of this study, we intended to map, through an online review, the 
interrelationship between the centers of value and the resource characteristics needed to fully 
describe them. The sites under study came from three sources: (1) those suggested to us by 
faculty through survey responses and discussions, (2) our own research, and (3) initiatives 
funded by the Mellon and Hewlett foundations. The list of sites can be found in Appendix A. 

As we began to collect these data, however, two issues became apparent: 1) the sheer scope of 
data available made collection both time consuming and resource intensive, and 2) the data 
themselves were in different formats, making comparison difficult at best. We opted to forego 
this mapping process at the same time as it became apparent that our planned broad survey 
(described below) would not sufficiently answer the questions we posed. 

Specifically, we thought it more valuable to tease out some of the complexities and richness of 
individual sites. Thus, we opted to conduct in-depth one-on-one telephone interviews with a 
small sample of sites in lieu of a broad survey that might only scratch the surface. Though not 
comprehensive of the entire digital resource universe, our interviews were able to explore the 
variety and the nuances of individual sites, and thus begin to raise important questions and 
identify areas for future investigation. Our findings are discussed in Goal 3A. 



77 http:/ / sunsite.berkeley.edu 
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GOAL 2: HOW ARE DIGITAL RESOURCES BEING USED AMONG DIVERSE 
COMMUNITIES? 

A major part of this project includes an analysis of how, how much, and even if unrestricted 
digital resources are being used in targeted H/SS teaching/ learning contexts among diverse 
higher education communities. We are taking two approaches to this challenge: First, we talked 
with faculty directly through discussion groups and surveys (Goal 2A), and second, we tested 
transaction log analysis (TLA) and online site surveys to determine their efficacy and efficiency 
as methods for understanding users of specific sites (Goal 2B). 

GOAL 2A: FACULTY DISCUSSION GROUPS AND FACULTY SURVEY 

Principal authors: Diane Harley, Jonathan Henke, Irene Perciali , and David Nasatir, 
with Ian Miller and Shannon Lawrence 

This section describes methods and results from conducting surveys and discussion groups 
with instructors (faculty, teaching assistants, and students) from UC Berkeley, other UCs, liberal 
arts colleges, and community colleges; the sample included both users and non-users of digital 
resources. Additionally, we conducted discussion groups with librarians, collection owners, 
and educational technology professionals. 78 We also relied on interviews, background research, 
and our own personal knowledge. 

A. Discussion groups - methods 

In October and November 2003, we hosted four discussion group sessions with thirty-one 
instructors from three institutions. Twenty-five faculty and six graduate student instructors 
participated. Transcripts of these discussion groups were analyzed. 79 The discussions formed 
the basis for the development and creation of the faculty survey instrument. 

We asked variants of four basic questions in our discussions: 

• What digital collections/ resources do you use in undergraduate teaching? 

• How do you use them in your teaching? 

• What obstacles to use do you encounter? 

• What would you like to do with digital collections/ resources in a perfect world? 

In summary, we found that instructors use a diverse array of online resources in their teaching. 
Those in our discussions generally agreed that there were many useful online resources 
available, but issues frequently arose about how to find, manage, maintain, and reuse them in 
new contexts. Resources mentioned run the gamut, and include the ubiquitous Google search, 
student-created "collections," archived news and other media sources, digital library 
collections, subject portals that link to other sites and resources, archived student discussions, 
blogs, and so on. Simply put, no two collections are alike, especially given the relevance and 



78 In April and May 2004 CSHE and the California Digital Library co-sponsored two meetings with educational 
technology professionals and librarians to discuss issues of use, available tools, and the potential for synergies among 
the multiple groups represented. 

79 The methodology and full discussion group analysis, plus the survey instrument, can be found at our project 
website: http:/ / digitalresourcestudy.berkeley.edu. 
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value individual faculty apply to a particular digital collection and its potential use in the 
classroom. 

Personal teaching style and philosophy influenced use significantly. We also found some 
intriguing differences among disciplines in their choice of and concerns about the digital 
resources they use. In some cases, faculty in disciplines that use texts extensively depend on 
different kinds of sources for different pedagogical goals than those who rely more heavily on 
images. Those instructors who used text in writing and composition courses sometimes turned 
to online texts as examples of good or bad writing or as exercises to teach students basic 
information literacy. Some faculty in fields dependent on visual analyses and / or current events 
(e.g., cultural studies, new media, art history, and anthropology) were heavy users of digital 
resources. Those faculty who teach subjects that require three-dimensional visualization 
and/ or historical reconstruction used various pedagogies that require students to explore and 
evaluate sites and associated evidence in both time and space. These pedagogies use a 
significant "problem-based inquiry" approach, and often integrate distributed online resources 
and/ or those hand-worked by the instructors. 

The tasks that faculty report having to accomplish, and for which they frequently do not have 
help, include: 

• Locating or searching for digital resources 

• Assessing the technical quality of digital resources 

• Assessing the credibility of digital resources 

• Evaluating copyright and fair use 

• Evaluating the appropriateness of resources for their teaching goals 

• Creating their own websites 

• Learning how to use learning management systems such as WebCT 

• Importing resources into a course website or database 

• Digitizing and posting resources to a webpage or database 

• Assembling and building collections 

• Curating or indexing the digital resources 

• Preserving digital collections and keeping them relevant 

• Finding short- and long-term storage for collections 

• Guiding students in how to find and evaluate digital resources 

• Securing resources such as servers and smart classrooms, which are often scarce. 

B. Faculty survey - methods 

Direct discussion with and feedback from faculty gave us a deeper understanding of why 
faculty do or do not use unrestricted resources in undergraduate teaching. Specifically, our 
analysis of the discussion groups identified four areas to inform our survey design and to 
explore in more detail with the survey responses and analyses: 

• Digital collections vary in type, purpose, and perceived value. 

• There is significant variation in faculty enthusiasm and involvement. 
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• Different disciplines have different needs. 

• There is a wide range of obstacles to using digital collections. 

Initially, we surveyed approximately 4,500 faculty from specific departments at a stratified 
random sample of community colleges. University of California campuses, and liberal arts 
colleges in California. 80 This survey was distributed in mid-March 2004, and had a closing date 
in June 2004. 

We then conducted a second survey of a larger population, to expand our pool of respondents 
beyond the targeted institutions and geographical areas (e.g., international, non-UC research 
universities, state colleges, etc.); we targeted subscribers of H-Net lists and the Humanist 
discussion group for this second survey. (See page 4-13, below.) 

1. Target population: Institutions and disciplines 

The primary target population for the initial survey consisted of active teaching faculty and 
instructors in selected H/SS disciplines and institutions in California. There were pronounced 
differences of size, geographic distribution, mission, student body composition, and faculty 
background among the types of institutions and the various disciplines (below). The 
population included three types of California universities and colleges: campuses of the 
University of California (UC), community colleges, and private non-religious liberal arts 
colleges in California. 

Out of necessity, we could not sample all possible disciplines for this initial scan of the universe. 
We surveyed well-established humanities and social science disciplines that, based on our 
research and general knowledge, were likely to have use for the types of unrestricted digital 
resources germane to this study. 

The population consisted of all levels of undergraduate classroom instructors in the disciplines 
of visual arts, art history, archaeology, architecture, anthropology, political science, history, 
languages and literature, writing and composition, classics, and geography; it excluded English 
as a second language (ESL), remedial English, and American Sign Language (ASL). 

Our research into individual schools and departments revealed that the chosen disciplines 
included faculty who were cross-listed in other programs (e.g.. Medieval Studies). Also, unlike 
specialized programs or certain interdisciplinary studies (such as Near Eastern Studies), these 
disciplines could be found at all three types of institutions sampled for this project. 

2. Sampling 

For this survey, we drew a stratified random sample of institutions from each of the specified 
institution types in California: UCs, community colleges, and private non-religious liberal arts 
colleges. (Within the community college system, stratification was also based on size and 
geographic region.) This design permitted pooling of a large number of faculty responses and 
examination of variations in digital collection usage associated with qualities of departments, 
disciplines and institutions, and attributes of individual users (e.g., academic rank). 



80 Because this survey was only one piece of a larger project and because of time limitations, we chose to exclude 
other institution types, such as the California State University system and private universities, from our sample. 
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At each institution, all active instructors (full- and part-time) who were currently teaching in 
any of the selected disciplinary areas were asked to participate in the survey. In summary, 
institutions in the sample include: 

a. University of California 

Of the eight eligible UC campuses with undergraduate instruction, we randomly selected five: 
UC Berkeley, UC Davis, UC Los Angeles, UC Santa Barbara, and UC San Diego. 

b. California community colleges 

Given the geographic and size differences among the state's more than 100 community colleges, 
we stratified our sample for this group by size (small, medium, and large) and by the 
population density of the areas they serve (urban, suburban, and rural). Classification data were 
provided by the California Community Colleges Chancellor's Office (1999). We then randomly 
chose three of each type of small school, two of each type of medium school, and one of each 
type of large school. Based on this method, we chose: 



Table 2.1: Community college sampling 



Type of School 


Small 

(< 10,000 students) 


Medium 

(10,000-15,000 students) 


Large 

(> 15,000 students) 


Rural 


(3): Mendocino College, 
Columbia College, and 
Porterville College 


(2): Antelope Valley College, 
Monterey Peninsula College 


(1): San Joaquin Delta College 


Suburban 


(3): Oxnard College, Marin 
College, and Santiago 
Canyon College 


(2): Foothill College, Mission 
College 


(1): Saddleback College 


Urban 


(3): LA Southwest College, 
College of the Alameda, LA 
Harbor College 


(2): Golden West College, LA 
Trade-Tech 


(1): East LA College 



c. Liberal arts colleges 

The Carnegie Classification defines liberal arts colleges as "primarily undergraduate colleges 
with major emphasis on baccalaureate programs, . . . which award at least half of their 
baccalaureate degrees in liberal arts fields." Carnegie lists fourteen liberal arts colleges in 
Calif ornia — one public and thirteen private. 81 In consultation with the Mellon Foundation, we 
limited our sample to private institutions without an explicit religious or ethnic mission. The 
final list included eight colleges: Claremont McKenna College, Harvey Mudd College, Mills 
College, Occidental College, Pitzer College, Pomona College, Scripps College, and Whittier 
College. We included all in our sample. 



81 http://www.carnegiefoundation.org/Classification/CIHE2000/PartIfiles/BA-LA.htm 
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3. Identifying participants and finding contact information 

A combination of public resources was employed to identify all current instructors within the 
selected disciplines at each sampled institution. We relied on university catalogs, online 
resources, and commercial databases to identify full- and part-time faculty as well as adjunct 
faculty and instructors. We also contracted with a marketing company to obtain an up-to-date 
database of faculty from our various institutions by discipline. 

A total of 4,488 faculty members were identified for participation. We identified 3,277 
participants from our own research into public domain sources; 3,249 were identified from the 
list provided by the marketing firm (with an overlap of 2,038 identified from both sources). 

After contacting potential participants, we found that a number of individuals were ineligible 
(retired, deceased, or not teaching undergraduates in the targeted disciplines). As a result, we 
had a total valid sample of 4,443 faculty members across all three institution types (Table 2.2, 
below). 

The contact lists were extensively cross-checked and cleaned, and additional research was 
performed to fill in missing contact information, to verify contact information, and to verify 
subject eligibility. We compiled contact information for selected instructors using a variety of 
public sources. Specifically, we obtained participant email addresses and work mail addresses 
via public online or written sources (such as directories, departmental websites, etc.) and/ or by 
requesting information that is publicly available but in a convenient, centralized format via 
department chairs or administrators and/or other university administrators. This latter process 
was necessary because (1) online sources are often out of date, incomplete, and/ or 
disorganized, and (2) departments and universities often have public but centralized lists of 
faculty work contact information. 

Valid email addresses were found for 3,596 participants; the remaining 892 were contacted by 
postal mail (Table 2.2). (After the initial recruitment email, 240 (6%) of the messages "bounced" 
because the email address was no longer valid.) 

The process of identifying survey subjects from the thirty-one targeted institutions and finding 
their contact information was incredibly long and labor-intensive — much more than we would 
have predicted. We spent approximately 1,100 staff hours over five months compiling and 
cleaning our list of survey subjects and contact information. Comparing our list (generated 
from our own research) with the list obtained from the marketing company confirmed the value 
of using both methods: there was surprisingly little overlap between the two, and we would 
have missed fully one-fourth of our sample if we had relied on either method alone. 

Our difficulty identifying current instructors was particularly acute in the community colleges. 
With a large number of part-time instructors and adjunct faculty, even the institutions 
themselves sometimes do not have a comprehensive list of their instructors. Even after 
identifying instructors' names, determining their email or postal addresses was no mean feat. 
Smaller and less well-funded institutions also seem less likely to have central email systems or 
good online directories. After investing considerable time and energy identifying and 
contacting our sample group of instructors and faculty members, we are convinced that there 
would be no simple, straightforward, reliable way of conducting a comprehensive survey of 
this population. 
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4. Delivery mode: Online vs. paper 

One major question in planning the survey was whether to administer the survey online or on 
paper (by mail). It is worth noting that the literature around online vs. paper-based surveys is 
unclear. We consulted extensively with UC Berkeley's Survey Research Center (SRC) and took 
a number of factors into consideration when making this choice: 

• Our research group had significant experience with online surveys, as well as other 
survey modes. 

• Recruitment, data collection, and data entry would be much simplified (and more cost- 
effective) if responses were entered online. 

• An online survey would likely bias responses toward those more comfortable with 
technology. 

• Recruitment for an online survey would be further complicated by our difficulty finding 
valid email addresses for many of our subjects, especially in the community colleges. 

We also considered what response rate we might expect with each of the two delivery methods. 
The SRC had recent experience with an institutional online survey of UC faculty that achieved a 
50% response rate after three email reminders. On the other hand, a 2001 report from the 
RAND Corporation reviewed online surveys and concluded that response rates were higher on 
paper-based surveys (Schonlau et al., 2001). 

Our own experience and discussion with others who have conducted online surveys suggest 
that other factors may determine response rate in academic settings. We see a number of issues 
which, in combination, may impact survey response rates: 

• Survey fatigue— e.g., the multiple surveys faculty are exposed to (e.g., the recent Mellon 
Foundation survey, Schonfeld and Guthrie, 2004, and various local surveys) 

• The source of the survey (institutional vs. outside surveys) 

• Whether the source of the survey is in a position of authority (e.g., a vice chancellor) 

• The use of incentives 

• The motivation of respondents. Some individuals may simply like to answer surveys, 
leading to over-sampling of this group. 

Because a major goal of the faculty survey is to explore the perspectives of people who do not 
use digital resources — including those faculty members who are less adept with computer 
technology — we decided not to administer the survey completely online. In order to achieve 
some of the benefits associated with online surveys, however, we decided to offer the survey 
online for instructors with valid email addresses available, and by mail for the remaining 
instructors. In addition, people who were initially recruited by email were given the option of 
requesting a paper version of the survey. This mixed-mode survey achieved some of the 
efficiencies associated with online surveys, while minimizing the potential response bias. 

5. Design of survey instrument 

We used the results of our faculty discussion groups (see above) to guide our design of the 
survey instrument. The questionnaire was also reviewed for refinement by experts at UC 
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Berkeley's Survey Research Center (SRC). The survey was designed to take fifteen to twenty 
minutes to complete, and focused on eight main topics: 

• Teaching background 

• Types and sources of digital resources used 

• Personal digital collections 

• How digital resources are used in teaching 

• Motivations for using digital resources 

• Motivations for not using digital resources 

• Barriers and frustrations 

• Support and assistance. 

(The complete survey instrument can be found in Appendix D and is also linked from 
http:/ / digitalresourcestudy.berkeley.edu/ docs.html.) 

6. Pilot testing 

In the spring of 2004, we conducted formal pilot tests of the survey instrument. Pilot tests were 
conducted with five participants: faculty members and instructors from several different 
disciplines and two different institutions. The pilot tests provided valuable feedback about the 
questionnaire; the survey was modified to improve its clarity, understandability and validity. 

7. Electronic survey system 

After assessing commercial options and our own data management and analysis needs, we 
decided to build a custom online survey system in-house for this project. The system supports 
the entire survey process, including delivery of the survey online, online data and survey 
management, management of the list of survey subjects, sending and tracking recruitment 
emails, and preliminary data analysis. It is a secure system; all survey transactions are 
encrypted using Secure Socket Layers (SSL), and data are exported in an encrypted format. 
Additionally, all identifying information is stored separately from survey responses. The 
system allows auditing and verification of all survey responses. The system is written in Perl 
with a Postgres database, and can be used for any online, web-based survey. 

8. Survey recruitment: Online and by mail 

When possible, faculty were recruited via email solicitations. The email contained a URL link to 
the online survey with a unique random identifying code for each respondent. Participants 
without email addresses (and those who requested a paper version) were sent paper copies of 
the survey by regular postal mail, along with a pre-paid response envelope. 

As an incentive to participate, all faculty members who completed the survey were entered into 
a prize drawing for a new Palm PDA, a $300 Amazon.com gift certificate, or one of five $100 
Amazon.com gift certificates. In addition, we offered all contacted faculty the opportunity to 
receive a report detailing our research findings at the end of the project. In addition to showing 
our good will and legitimacy, we attempted to establish a sense of reciprocity that would 
encourage faculty to participate. 
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For people contacted by email, two follow-up efforts were made to increase the number of 
responses. For those contacted by mail, a reminder postcard was sent several weeks later. 

We also conducted a short telephone survey of a sample of non-responders, to explore their 
reasons for not responding and to look for possible sources of response bias. (See page 4-11, 
below.) 

9. Human subjects and confidentiality 

This project was approved by all the relevant institutional review boards (IRBs), including the 
UC Berkeley Committee for the Protection of Human Subjects. All measures were used to 
protect any identifiable data (e.g., each participant has been assigned an identification code; all 
responses and code keys were stored separately in locked files or on secured computers). No 
participants will be identified in any reports of the research. All research staff with access to 
survey data successfully completed the NIH tutorial "Human Participant Protections: 

Education for Research Teams" to fulfill the obligation for education in the area of human 
participant protection. 82 The research team was also certified by the UC Santa Barbara Office of 
Human Research Protections. 

Given the decentralization of the UC institutional review boards, we were also required to 
apply for human subjects' permission at UC Davis and UC Santa Barbara. UC Los Angeles 
accepted our UC Berkeley approval for an exemption and did not require a separate IRB review. 
UC San Diego does not require a separate IRB review if faculty information is obtained through 
public sources rather than university administrators, and if administrators are not involved in 
recruiting faculty. We obtained approval using an exempt protocol for UC Davis and a full 
protocol for UC Santa Barbara. Both institutions required a local principal investigator (PI). 
Barbara Sommer, UCD, and Richard Flacks, UCSB, joined us as Pis for their respective 
campuses. 

10. Data cleaning and calculated variables 

After compiling the complete response data set, we performed basic data cleaning. We began 
by validating respondents' identities, to be sure that respondents were actually part of the 
sample pool and to remove responses from people who were clearly ineligible for study 
participation (e.g., people who do not teach undergraduates or are not in our target disciplines 
or institutions). We also corrected obvious data-entry errors and validated responses of "other" 
(questions 1, 5, 6, 7, 10, 17, 18, 19, 22, 23, and 25) to see if they were better classified under one of 
the existing, closed-ended categories. 

The survey question about disciplines (question 1, see Appendix D) asked respondents to 
"check all that apply"; 334 instructors (40%) selected more than one discipline. For analyses 
that required mutually exclusive disciplinary categories, we recoded respondents into eleven 
disciplinary categories based on their responses to question 1, their free-text descriptions of 
their disciplines in question 1, and the specific courses they taught (question 2). These 
disciplinary categories are shown in Table 2.3 below. (Twenty-four respondents could not be 
categorized, due to missing or ambiguous data.) 



82 http:/ /cme.cancer.gov/c01/nih_intro_01.htm 
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We created binary recodes of most of the scaled response variables, to simplify some analyses. 
For frequency scales (five-point scale; questions 5, 6, 7, 9, 10, and 21), we coded responses of 
"sometimes" or greater as "yes"; for agreement scales (four-point scale; questions 12, 13, 14, 15, 
16, 17), we used "somewhat agree" and "strongly agree"; for importance scales (five-point scale; 
question 18), we used "somewhat important" or greater; for satisfaction scales (four-point scale; 
question 19), we used "somewhat satisfied" or "extremely satisfied." Most tables and figures in 
the results section below use these binary recodes. 

We also constructed a composite index reflecting each instructor's total use of digital resources 
in undergraduate teaching. The index was based on respondents' reports of the relative 
frequency with which they used seventeen individual types of resources (survey question 5; see 
Appendix D). The index was then scaled from zero to 100, with a score of zero indicating no 
usage of any resource type and a score of 100 indicating maximum usage of all seventeen 
resources. Among survey responders, the use index had a mean score of 31 and a standard 
deviation of 16. Figure 2.1 shows the distribution of index scores, with a normal distribution 
superimposed. 

Figure 2.1: Index of total digital resource use, distribution 

30 1 1 
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Use Index Score 

11. Response rates 

We received 831 valid responses to the survey, of which 729 completed the entire survey and 
102 began the survey but did not finish. (An additional 115 people viewed the online survey, 
but did not answer any questions.) Responses by institution type are shown in Table 2.2. 
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Table 2.2: Survey recruitment and response rates 





Total 


University of 
California 


Liberal arts 
colleges 


Community 

colleges 


Total: 










Eligible participants 


4,443 


2,416 


428 


1,582 


Submitted surveys 


831 


522 


88 


209 


Response rate 


19% 


22% 


21% 


13% 


Online Survey: 










Participants contacted 


3,553 


2,240 


397 


900 


Submitted surveys 


771 


509 


85 


165 


Response rate 


22% 


23% 


21% 


18% 


Paper survey: 










Participants contacted 


889 


176 


31 


682 


Submitted surveys 


60 


13 


3 


44 


Response rate 


6.8% 


7.4% 


9.7% 


6.5% 



We note that people recruited by email responded at three times the rate of those recruited by 
postal mail. (This could be due to the email reminder messages. 83 ) Within each delivery mode, 
the response rates were similar across the three institution types, with community colleges 
slightly trailing UCs and liberal arts colleges. However, because so many more community 
college instructors were recruited by postal mail, the overall response rate at those institutions 
was one-third lower than at UCs and liberal arts colleges. 

Table2.3 presents a demographic description of the survey respondents. 



83 We observed a greater response volume following each reminder message than following the initial recruitment 
message. 
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Table 2.3: Demographics of faculty survey respondents 





Total 


University of 
California 


Liberal arts 
colleges 


Community 

colleges 


Full-time 


78% 


86% 


88% 


56% 


Part-time 


22% 


14% 


12% 


44% 


Highest Degree: 










AA 


0.1% 


0% 


0% 


0.6% 


BA 


2.4% 


1.7% 


1.2% 


5.0% 


MA 


26% 


11% 


11% 


71% 


PhD 


69% 


86% 


87% 


19% 


MD 


0.5% 


0.2% 


1.2% 


1.1% 


JD 


0.9% 


0.8% 


0% 


1.7% 


Other 


0.4% 


0.2% 


0% 


1.1% 


Gender: 










Male 


49% 


55% 


48% 


34% 


Female 


51% 


45% 


52% 


66% 


Age (years): 










0-29 


1.0% 


0.9% 


0% 


1.7% 


30-39 


19% 


18% 


27% 


16% 


40-49 


27% 


29% 


32% 


19% 


50-59 


32% 


33% 


28% 


32% 


60-69 


18% 


17% 


12% 


25% | 


70- 


3.4% 


2.8% 


1.2% 


6.3% 


Discipline: 










Anthropology and archaeology 


8% 


8% 


8% 


5% 


Art and architecture 


12% 


11% 


12% 


15% 


History 


12% 


13% 


18% 


8% 


Political science 


11% 


12% 


19% 


4% 


Writing 


4% 


4% 


3% 


3% 


Foreign language 


11% 


12% 


10% 


11% 


Literature and English 


28% 


25% 


24% 


40% 


Geography 


2% 


3% 


0% 


2% 


Ethnic, gender, and cultural studies 


3% 


4% 


2% 


1% 


Media studies & communications 


2% 


1% 


2% 


2% 


Other 


4% 


4% 


0% 


5% 



12. Non-response survey 

To assess the degree to which faculty survey respondents were representative, we conducted a 
follow-up telephone survey of a random sample of survey non-respondents. The non-response 
sample was recruited and interviewed by UC Berkeley's Survey Research Center. 

We successfully surveyed 128 non-respondents, distributed among the three institution types. 
The overall response rate on the follow-up survey was 76 percent (see Table 2.4). 



The non-response survey instrument can be found in Appendix E. The survey required about 
five minutes and covered three main areas: 



• Reasons for non-response (to estimate whether their reasons were likely to bias the 
primary findings of the survey) 

• Demographic measures — age, gender, discipline, highest degree obtained, and length 
of time since highest degree (to look for demographic bias) 
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• Five sample questions from the main survey, which were closely correlated with overall 
use of digital resources (to compare the non-respondents with the respondents and look 
for potential bias). 



In addition, we knew the gender and discipline of all eligible subjects. 



Table 2.4: Non-response survey response rates 





University of 
California 


Liberal arts 
colleges 


Community 

colleges 


Total 


Total attempted 


63 


63 


63 


189 


Ineligible 


5 


6 


9 


20 




(7.9%) 


(9.6%) 


(14.3%) 




Total eligible 


58 


57 


54 


169 


Non-response 










Never located 


7 


6 


2 


15 


Never home 


7 


5 


7 


19 


Refused 


3 


3 


1 


7 


Total non-response 


17 


14 


10 


41 


(% of eligible) 


(29.3%) 


(24.6%) 


(18.5%) 


(24.3%) 


Response Rate 


41 


43 


44 


128 


(% of eligible) 


(70.7%) 


(75.4%) 


(81.5%) 


(75.7%) 


(% of eligible successful contacts) 


(93.2%) 


(93.5%) 


(97.8%) 


(94.8%) 



a. Non-response survey results 

Analysis of the demographics shows that respondents and non-respondents were fairly similar. 
Statistically significant findings are summarized in Table 2.5, and complete results can be found 
in Appendix F. Overall, there were no statistically significant differences between the two 
groups in age or education level, or gender, based on the non-response survey. The only 
statistically significant difference overall was the under-representation of art practice 
instructors. 

We also analyzed response patterns separately for the three different institutional types. 
University of California: 

• Although most differences were not statistically significant, differences by discipline are 
notable: foreign language, art history, and political science were slightly 
overrepresented; architecture, art practice or applied arts, geography, and history were 
slightly underrepresented. 

• Women were slightly overrepresented. 

• There was no bias by age or education level achieved. 



Liberal arts colleges: 

• There was some bias by age: on average, respondents were five years younger than 
non-respondents . 

• There was no bias by discipline, gender, or education. 



Community colleges: 
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• There was some bias by gender, with women slightly overrepresented. 

• There was no bias by discipline, age, or education. 



Table 2.5: Non-response survey: Summary of differences 

(Includes only differences that were statistically significant at the 0.05 level.) 





Institution type 


All 


University of California 


Liberal arts colleges 


Community colleges 


Demographic measure 


over 


under 


over 


under 


over 


under 


over 


under 


Gender 














Women 


Men 


Age 










Younger 


Older 






Education 


















Discipline 




Art 

practice 




History 




Literature, 

classics 




Art 

practice 


Usage level 


















Enthusiasm 



















Based on responses to the five key survey questions, overall usage of and enthusiasm about 
digital resources can be estimated for non-respondents and respondents. On these measures, 
the non-respondents seem very similar to the survey respondents, both overall and within each 
institution type, with overall means and standard deviations almost identical and no significant 
difference detected. This suggests that although there may be slight differences in the 
demographic characteristics of the respondents and non-respondents, these characteristics are 
unrelated to the key areas of interest of the study. 

Reasons given by non-respondents for failing to complete the original survey might be expected 
to reveal sources of non-response bias. In fact, however, very few people indicated usage- 
related reasons that might suggest bias on the study's key topics. 84 Instead, the majority cited 
reasons such as not having received the initial invitation, forgetfulness, or lack of time. 

Taken together, the small demographic differences, the similarity between the groups on the 
sample survey questions, and the reasons given for non-response suggest that the survey 
respondents may be considered reasonably representative of the population in the areas of 
primary concern to this study. We believe that the findings based on this survey may be 
cautiously applied to the whole population of instructors in the targeted disciplines at these 
three institution types in California. 

13. H-Net survey 

As a follow-up to the California faculty survey, we conducted a second, parallel survey of a 
broader population in fall 2004. This second survey was targeted at college and university 



84 Reasons such as lack of knowledge about digital resources, opposition to the use of digital resources in teaching, or 
inapplicability of digital resources to their discipline or teaching style. 
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instructors in the humanities and social sciences, both domestically and internationally; the 
survey was available on the Internet for any instructor to complete anonymously. 

The survey instrument was largely identical to the instrument from the California faculty 
survey, with a few alterations to account for international audiences and instructors from a 
broader range of institutions and H/SS disciplines. We solicited participation by posting 
announcements on online discussion boards and email lists — primarily the various lists 
associated with H-Net, an online community of scholars and teachers in the humanities and 
social sciences. Announcements were also re-posted on other related online lists and forums, 
such as the Humanist discussion group. 

Because the survey was not sampled from a defined population, there was no response rate as 
such and no way to determine if the respondents were representative. In fact, we fully 
anticipated that the responses would not be representative of instructors in general, but would 
reflect the greater awareness, use, and enthusiasm for digital resources that is typical of online 
communities. 

a. H-Net respondent demographics 

We received responses from 452 subjects for this second survey, of which 353 (78%) completed 
the survey and the remaining 99 (22%) discontinued participation before the end. The 
respondent demographics differed somewhat from the survey of California instructors, as 
shown in Table 2.6, below. Respondents were predominantly from four-year colleges and 
universities (with over half from research universities) in North America and Europe. We 
received a small number of responses from Australia, Asia, and the Middle East, and none from 
Latin America or Africa. As anticipated, respondents represented a broad range of disciplines, 
but a disproportionate number (40%) were history instructors — perhaps because of a high 
number of history instructors on the email lists used to publicize the survey. Respondents from 
this survey tended to be younger than those from the California faculty survey and had 
completed their education more recently. 



Table 2.6: H-Net survey and faculty survey respondent demographics 





International (H-Net) survey 


California faculty survey j 


N 


Percent 


N 


Percent 


Total 


452 


100% 


831 


100% 


Discipline: 










Languages and literature 










English 


— 


— 


182 


22% 


Languages 


32 


7.1% 


149 


18% 


Literature 


41 


9.1% 


182 


22% 


Speech/ rhetoric 


17 


3.8% 


- 


- 


Writing/composition 


51 


11% 


165 


20% 


Humanities/ social sciences 










Anthropology/ sociology 


53 


12% 


62 


7.5% 


Archaeology 


20 


4.4% 


24 


2.9% 
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International (H-Net) survey 


California faculty survey [ 


N 


Percent 


N 


Percent 


Architecture 


5 


1.1% 


20 


2.4% 


Classics 


11 


2.4% 


19 


2.3% 


Communications (media/film studies) 


22 


4.9% 


- 


- 


Cultural studies (ethnic/ gender studies) 


64 


14% 


- 


- 


Economics 


9 


2.0% 


- 


- 


Geography 


13 


2.9% 


24 


2.9% 


History 


190 


42% 


128 


15% 


International studies 


13 


2.9% 


- 


- 


Legal Studies/law 


5 


1.1% 


- 


- 


Linguistics 


5 


1.1% 


- 


- 


Philosophy 


23 


5.1% 


— 


— 


Political science/ public 
policy / government 


47 


10% 


92 


11% 


Art history 


43 


9.5% 


47 


5.7% 


Art practice/visual art/ other art 


9 


2.0% 


63 


7.6% 


Performing arts (music, theater, dance) 


9 


2.0% 


- 


- 


Psychology 


6 


1.3% 


— 


— 


Religious studies 


34 


7.5% 


- 


- 


Other 


39 


8.6% 


157 


19% 


Geography: 










Africa 


0 


0% 


— 


— 


Asia or Pacific Islands 


5 


1.4% 


- 


- 


Australia 


4 


1.1% 


- 


- 


Canada 


16 


4.5% 


- 


- 


Central America 


0 


0% 


- 


- 


Europe 


30 


8.4% 


- 


- 


Mexico 


0 


0% 


— 


— 


Middle East 


3 


0.8% 


- 


- 


South America 


0 


0% 


- 


- 


United States 


300 


84% 


831 


100% 


Institution Type 










University of California 


- 


- 


522 


63% 


Liberal arts college 


- 


- 


90 


11% 


Community college 


— 


— 


206 


25% 


Institution Type 










Research university, public or private 
(Ph.D. granting) 


182 


51% 


— 


— 


Public (state) university or college (non- 
Ph.D.) 


67 


19% 


— 


— 


Private university or college (non-Ph.D.) 


77 


22% 


- 


- 


Community college (two-year, 
associates degree) 


25 


7% 


— 


— 


Professional school/ college (post- 
graduate only) 


0 


0% 


— 


— 


Technical or vocational institution 


2 


0.6% 


- 


- 


Secondary school (high school or 
middle school) 


1 


0.3% 


— 


— 


Primary school 


0 


0% 


- 


- 


Other 


3 


0.8% 


13 


1.6% 


Institution Size 










0 - 1,000 students 


15 


4.2% 


- 


- 


1,000 - 5,000 students 


85 


24% 


- 


- 


5,000 - 10,000 students 


68 


19% 


- 


- 


10,000 - 25,000 students 


96 


27% 


- 


- 


more than 25,000 students 


90 


25% 


-- 


-- 
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International (H-Net) survey 


California faculty survey [ 




N 


Percent 


N 


Percent 


Job Title 










Instructor 


28 


7.9% 


64 


8.6% 


Lecturer 


39 


11% 


124 


17% 


Adjunct professor 


33 


9.3% 


59 


7.9% 


Assistant professor 


82 


23% 


100 


13% 


Associate professor 


57 


16% 


126 


17% 


Professor 


57 


16% 


255 


34% 


Retired or emeritus professor 


1 


0.3% 


10 


1.3% 


Graduate student as main instructor 


25 


7.1% 


4 


0.5% 


Graduate student as teaching assistant 


19 


5.4% 


- 


- 


Secondary school (middle/ high school) 
teacher 


1 


0.3% 


— 


— 


Primary (elementary) school teacher 


0 


0% 


- 


- 


Other 


12 


3.4% 


6 


0.8% 


Part-time 


125 


35% 


161 


22% 


Full-time 


228 


65% 


584 


78% 


Highest degree 










Associate's degree (two-year 
undergraduate degree) 


0 


0% 


1 


0.1% 


Baccalaureate (bachelors) degree (four- 
year undergraduate degree) 


10 


2.8% 


18 


2.4% 


Master's degree (post-baccalaureate) 


107 


30% 


198 


26% 


Ph.D. (Doctor of Philosophy) 


235 


66% 


519 


69% 


M.D. (Medical Doctor) 


- 


- 


4 


0.5% 


J.D. (Doctor of Law) 


0 


0% 


7 


0.9% 


Other doctoral degree 


3 


0.8% 


- 


- 


Other degree 


1 


0.3% 


3 


0.4% 


Age (years) 










00-35 


98 


28% 


76 


10% 


36-44 


94 


27% 


169 


23% 


45-53 


84 


24% 


181 


25% 


54-61 


49 


14% 


184 


25% 


62+ 


20 


5.8% 


120 


16% 


Mean age ± SD (years) 


345 


44 ±11 


730 


50 ±11 


Mean years since degree ± SD (years) 


348 


10 ± 9.9 


734 


18 ± 12 


Gender 

Male 






362 


49% 


Female 


-- 


- 


379 


51% 



14. Principal component analysis 

Principal component analysis is a tool for simplifying datasets with a large number of 
quantitative variables. Principal component analysis derives a small number of linear 
combinations, or principal components, from a set of variables while keeping as much 
information from the original variables as possible. When there are many variables, principal 
components are an appropriate way to reduce the number of variables for clustering. 
Component matrices may be rotated to any number of equivalent matrices, with equal 
explanatory power. If rotations result in different interpretations, all interpretations are valid 
ways of describing the same thing (Pearson, 1901; and Rao, 1964). Note that principal 
component analysis assumes the input variables are continuous or interval variables, are 
normally distributed, and have equal variance at each point in the distribution. We felt that 
some violations of these assumptions might be warranted, as the results were to be used 
primarily as a suggestion. 



http:/ / digitalresourcestudy.berkeley.edu 



4-16 














Use and Users of Digital Resources — Harley et at./ CSHE/UCB 



April 2006 



To simplify the faculty survey data and to identify a smaller number of underlying components, 
we performed principal component analysis on five sets of questions. For each of these 
components, we then calculated component scores for each survey respondent. For the five sets 
of questions (ninety-eight raw variables), we produced twenty-two principal components;. 
Components were calculated for the types of digital resources used (twenty-six variables and 
five components), ways digital resources are used (nine variables and four components), 
reasons for using or not using digital resources (thirty variables and eight components), barriers 
to digital resource use (twenty-one variables and five components), and activities for which 
support is required (twelve variables and two components). 

Initially, we used the eigenvalue criterion to suggest the number of principal components to 
calculate; we adjusted these numbers upward or downward to produce components that were 
easier to interpret. We then performed an orthogonal varimax rotation, which maximizes the 
variance in the components. We assigned a name and description to each component, based on 
the individual variables with which each component was most strongly correlated. The 
component matrices can be found in Appendix G. 

Because each component score is calculated as the linear combination of a larger set of raw 
variables, any missing values in the raw variables makes it impossible to calculate all related 
component scores. To accommodate the missing values in our dataset and still produce a 
usable number of component scores, in cases where a small fraction (about 15 percent) of the 
raw data elements necessary to calculate a component score were missing, we filled in the mean 
response values for those missing variables, for component scoring. (If more than about 15 
percent of the data elements were missing, the component score remained missing.) 

a. Factor descriptions 

We calculated factor scores for five different sets of questions: 

• What digital resources people use (five factors) 

• How people use digital resources (four factors) 

• Reasons for using or not using resources (eight factors) 

• Barriers to digital resource use (five factors) 

• Activities with which people need support (two factors) 

What digital resources people use 

(1) General-purpose and reference materials - Including portals, reference resources, 
materials from search engines, exhibits, digital libraries, journals, and media sites. 

(2) Images and audiovisual materials - Including images, digital film or video, digital audio, 
simulations, and animations. Materials come from many sources, including commercial 
image databases, free image databases, and (occasionally) campus image databases. 

(3) Historical documents, maps, and primary sources - Including maps, facsimiles of 
historical manuscripts, images, and (occasionally) other texts or documents. Less likely to 
use news and media resources, blogs, and curricular materials. 

(4) Data, news/media, and governmental resources - Resources include datasets, 
governmental documents, and news resources (and occasionally maps). 
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(5) Discussion and curricular materials - Including blogs, class discussions, curricular 
materials, and digital readers/coursepacks. 

How people use digital resources 

(6) Student assignments - Assigned to students for projects, portfolios, or studying. 

(7) Web posting - Made available to students online. 

(8) Online courses - Presented in an online lecture or discussion. 

(9) In-class presentation - Presented in the classroom. 

Reasons for using or not using digital resources 

(10) Pedagogical reasons - A variety of reasons relating to teaching and learning, including 
providing students context and getting them excited, helping students learn and enabling 
their creativity, and allowing them to do more and have access to materials otherwise 
unavailable. 

(11) Expectations & reputation - Reasons relating to the expectations of students, colleagues, 
and administrators, and the instructor's associated professional reputation. 

(12) Inappropriateness - Reasons related to the inappropriateness of digital materials for 
teaching in general, or for the instructor's discipline or teaching style (and the instructor's 
lack of time to integrate them). 

(13) Concerns about students' interpretation and information literacy - Reasons related to 
students' lack of information literacy, and misuse (plagiarism) or decontextualization of 
digital materials. 

(14) Time, convenience, and access - Reasons related to improved efficiency and convenience 
for the instructor and students. 

(15) Teaching information literacy and critical thinking - Reasons related to the teaching of 
information literacy, critical thinking, and the evaluation of scholarship. 

(16) Making information publicly available - Concerns about making information publicly 
available to students and the general public, including allowing students to preview the 
course, making course materials available to the world, and integrating the instructor's 
research interests into the course. 

(17) Using free and publicly available materials - A dependence on resources that are freely 
or publicly available (and do not require passwords or registration). 

Barriers to digital resource use 

(18) Finding resources - Difficulty in finding appropriate resources, generally because there is 
too much relevant information available; includes a sense of being overwhelmed at the 
amount of material available, difficulty organizing materials, and the challenge of 
assessing documents' credibility and copyright status. 

(19) High-end multimedia equipment and software - Lack of easily available high-end 
equipment and software (which might typically be provided by the institution), including 
appropriate software for presentation and management of audiovisual materials, as well 
as servers, scanners, learning management systems, and classroom technology. 

(20) Personal equipment - Lack of easily available personal equipment for the instructor, 
including a computer, a high-speed connection, scanners, server space, and classroom 
technology. 

(21) Resource availability or existence - A belief that the materials an instructor needs simply 
do not exist, or are of poor academic quality. 
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(22) Student equipment - Lack of easily available equipment for the student, including 
computers and high-speed connections. 

Activities with which people need support 

(23) Technical activities - Activities primarily concerned with the technical support 
infrastructure, including setting up web pages or learning management systems, 
importing content, digitizing and organizing materials, obtaining and setting up 
infrastructure, and training students. 

(24) Intellectual and content-based activities - Activities primarily concerned with the digital 
materials themselves, including finding materials and assessing their appropriateness, 
credibility, and copyright status. 
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C. Faculty survey and discussion group results 

In analyzing the faculty surveys and discussion groups, we focused on five overarching 
questions: 

• What digital resources do you use in undergraduate teaching? 

• How do you use them? 

• What are your motivations for use (or non-use)? 

• What obstacles do you encounter? 

• In a perfect world, what would you do with digital resources? 

It is important to note that we assiduously avoided judgments about the "value" of specific 
resources. Instead we simply asked instructors to tell us what resources they found more and 
less useful, why or why not, and for what purposes. 

This section outlines in great detail the results and highlights of multiple analyses of the data 
sets. Our goal was to explore in as many ways as possible any emergent patterns in faculty use 
and attitudes. The results are presented in the following order: 

• Faculty survey data presented in the aggregate and by institution type for each group of 
questions: what is used, where it is found, how it is used, why it is used or not used, 
inhibiting factors to use, and support needs and satisfaction 

• Disciplinary variation based on principal component analyses 

• Responses by age 

• Variation among community colleges by size and population density 

• Differences between heavy and light users 

• Personal collections 

• Highlights from the H-Net survey 

• User profiling and cluster analysis. 

We reference the discussion group findings throughout when appropriate. 85 An analysis of the 
community college data by size and geography of institution is presented in Appendix J. 
Selected tables and figures are included in this report; additional data tables can be found in 
Appendix H. 86 

1. Faculty survey results: Aggregate and by institution type 

a. Types of resources faculty say they use 

As illustrated in Table 2.7 and Figure 2.2, the digitized resources faculty report using are not 
just text and do not just emanate from libraries or any other single source. Digital resources 
take many forms, and are found in media sites such as that of The New York Times, and in 



85 A complete summary of discussion groups is available on our website at 
http:/ / digitalresourcestudy.berkeley.edu/docs.html. 

86 A more complete set of data can be found online at http://digitalresourcestudy.berkeley.edu/surveyresults/. 



http:/ / digitalresourcestudy.berkeley.edu 



4-20 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



collections assembled by independent organizations and by individual scholars. According to 
our survey, the most commonly used resources (in approximate order of frequency of use) are 
images and visual materials, news and other media sources, online reference resources, digital 
film or videos, and portals. Curricular materials, blogs, digital coursepacks, and simulations 
were low on the list of resources that faculty in our sample say they use frequently (Figure 2.2). 

Media types encompass images, videos, audio recordings, news resources, digitized books, 
journal articles, data archives, maps, historical documents, curricular materials, and others. We 
were struck by how many faculty say they use video and audio, news, and other media 
sources. 87 

Among the three institution types, community colleges tend to be the heaviest users of digital 
resources, with the exception of digitized documents (where UCs and liberal arts colleges are 
higher), and government documents (where liberal arts colleges are higher). Liberal arts 
colleges use digital readers or coursepacks more than twice as heavily as UCs and community 
colleges. UC instructors are less likely to use simulations, audio, news and media resources, 
and online reference materials. 

Specific resources used: Open-ended survey question 8 asked respondents to list the specific 
digital resources or collections that they found most valuable for their teaching. Six hundred 
twenty-four people answered this question; they listed 574 unique responses, of which 408 were 
specific identifiable online resources. (The remainder included more general descriptions such 
as "online grammar websites" or "news online.") Detailed results can be found in Appendix I. 
There was little unanimity in faculty responses. Only ten websites were listed by more than 2 
percent of respondents (JSTOR, Google, Library of Congress American Memory Project, Library 
of Congress, Lexis Nexis, The New York Times, Modern Language Association, Perseus Digital 
Libraries, Oxford English Dictionary online, and Project Muse). We should note that three of 
these — Google, The New York Times, and JSTOR — were named in the survey instrument as 
examples; this fact may have artificially inflated their totals by bringing them to the 
respondents' attention (although these sites were heavily referenced in faculty discussion 
groups). 

These findings underscore our sense of the broad array of digital resources available to and 
used by faculty, and of the great variation in faculty choice of material to use. We were also 
struck by the frequent use of sites not specifically designed for educational purposes (including 
a wide variety of news, current events, governmental, and reference resources). Clearly, no one 
site has a monopoly on the educational resource domain. 



87 It should be noted that at least thirty-six free-text responses indicated some ambiguity about the distinction 
between offline media (e.g., VHS videotapes, CDs, or DVDs) and online digital resources. Therefore it is possible that 
some respondents who reported using digital audio and video may actually using non-web-based audio and video 
media rather than online digital audio and video. Our data do not allow a more fine-grained analysis. 
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Table 2.7: Faculty survey responses: Types of digital resources used 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How often do you use the following types of digital resources 
in your undergraduate teaching: 










Images or visual materials 


75% 


72% 


71% 


80% 


News or other media sources and archives 


64% 


55% 


74% 


81% 


"Portals" that provide links or URLs relevant to particular 
disciplinary topics 


63% 


61% 


68% 


66% 


Online reference resources 


62% 


57% 


67% 


72% 


Digital film or video 


62% 


57% 


66% 


72% 


Maps 


53% 


52% 


53% 


51% 


Online or digitized documents 


50% 


52% 


60% 


43% 


Audio materials 


46% 


41% 


46% 


54% 


Curricular materials and websites that are created by other 
faculty and/or other institutions 


35% 


32% 


36% 


43% 


Digital readers or coursepacks 


30% 


29% 


59% 


20% 


Online class discussions 


28% 


29% 


30% 


27% 


Government documents 


27% 


23% 


38% 


35% 


Data archives 


27% 


24% 


27% 


31% 


Digital facsimiles of ancient or historical manuscripts 


23% 


24% 


19% 


20% 


Simulations or animations 


19% 


15% 


17% 


29% 


Personal online diaries 


9% 


7% 


8% 


12% 
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Figure 2.2: Faculty survey responses by institution-type: What digital resources do you use? 



% of respondents who use each resource (at least sometimes): 



□ University of California 

□ Liberal arts colleges 
■ Community colleges 

Images or visual materials 

News or other media sources and archives 

"Portals" that provide links or URLs relevant to particular 
disciplinary topics 

Online reference resources 
Digital film or video 
Maps 

Online or digitized documents 
Audio materials 

Curricular materials and websites that are created by other 
faculty and/ or other institutions 

Digital readers or coursepacks 
Online class discussions 
Government documents 
Data archives 

Digital facsimiles of ancient or historical manuscripts 
Simulations or animations 
Personal online diaries 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
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b. Where people find resources 

As Table 2.8 indicates, faculty in our sample, like students, say they most commonly use 
Google-type search engines to find resources, including images. Another prominent source is 
faculty's personal collections. Online journals and public/free image databases are also high on 
the list of preferred ways of locating desired resources. Commercial image databases are very 
low on the list of sources, despite the costs to institutions in supporting them. Of course, many 
respondents may not realize that what they use is indeed licensed and they therefore may have 
misunderstood this question. 



Faculty at all three institution types were relatively heavy users of search engines, personal 
collections, public image databases, portals, library collections, and media sites, and they were 
all low users of commercial image databases, campus image databases, and online exhibits. 
Liberal arts college faculty were the heaviest users of online journals and library collections, 
while community college faculty were the heaviest users of media sites and the lightest users of 
online journals. 



Table 2.8: Faculty survey responses: Where faculty find digital resources 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How often do you use digital resources in your undergraduate 
teaching from each of the following sources? 










Search engines/ directories 


81% 


77% 


88% 


87% 


My own personal collection of digital materials 


69% 


69% 


64% 


71% 


Online journals 


62% 


65% 


80% 


46% 


Public online image databases 


62% 


62% 


60% 


62% 


Media sites 


57% 


52% 


60% 


69% 


Library collections 


57% 


57% 


70% 


52% 


"Portals" that provide links or URLs relevant to particular 
disciplinary topics 


55% 


53% 


59% 


58% 


Online exhibits 


37% 


36% 


43% 


37% 


Campus image databases from my own institution 


24% 


21% 


31% 


26% 


Commercial image databases 


9% 


5% 


10% 


15% 



c. Curricular materials 

Given the open educational resource movement to post freely available online curricular 
materials (e.g., MIT OCW), we highlight here faculty responses to the curricular materials 
question: "Please indicate how often you use or have used . . . curricular materials and websites 
that are created by other faculty and/ or other institutions (e.g., MIT OpenCourseWare, World 
Lecture Hall, Merlot)." 



When viewed in aggregate (see Table 2.7, above), use of electronic curricular materials was 
relatively low (when compared to news, images, personal collections, etc.) at 35%. When 
broken out by institution type however, it appears that community college faculty are heavier 
users (43% vs. 32% at UCs). Among the various disciplines (see Table 2.9, below), foreign 
language, writing, art and architecture, and geography faculty are the heaviest users of 
curricular materials, and anthropology, literature and language, history, and political science 
faculty are the lowest. These findings suggest that the former disciplines may be more 



http:/ / digitalresourcestudy.berkeley.edu 



4-24 







Use and Users of Digital Resources — Harley et at./ CSHE/UCB 



April 2006 



amenable to adopting digital curricular materials, and the latter more interested in customized 
products. 88 

It should be noted this survey question may not have captured the full range of electronic 
curricular use available to and used by faculty. For example, nineteen faculty wrote that they 
used textbook electronic supplements, although we did not ask explicitly about those types of 
resources. 

Table 2.9: Faculty survey responses: Use of curricular materials by discipline 





N 


Percent 


All 


831 


35% 


Media studies & communications 


13 


50% 


Foreign language 


93 


49% 


Writing 


33 


47% 


Art & Architecture 


102 


46% 


Geography 


18 


44% 


Anthropology & Archaeology 


65 


34% 


Literature & English language 


235 


31% 


Ethnic, gender, and cultural studies 


26 


29% 


History 


99 


28% 


Political science 


90 


21% 



d. How are resources used in teaching? 

Resources are used most frequently for presentation during class. The next most frequent uses 
were assigning resources to students for study or research projects and posting them on course 
websites. Very few instructors reported using digital materials for online courses or 
discussions, although this practice is more common at community colleges. 



It also appears that community college faculty are less likely to have a course website from 
which to post and link resources, perhaps due to lower institutional support or funding. 



Table 2.10: Faculty survey: How digital resources are used in teaching 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How often do you use digital resources in each of these ways? 










Presented during my lectures/ class 


71% 


68% 


72% 


78% 


Assigned to students for review and/or study 


59% 


56% 


65% 


64% 


Assigned for student research projects or problem-based learning 
assignments 


56% 


50% 


68% 


65% 


Posted directly on my course website 


52% 


57% 


48% 


38% 


Linked from my course website 


49% 


53% 


48% 


39% 


Used in tests and quizzes 


25% 


20% 


24% 


37% 


Assigned to students to create their own digital portfolios and/or 
multimedia projects 


20% 


18% 


27% 


22% 


Presented in the context of an online discussion 


12% 


9% 


11% 


21% 


Presented in my online lectures 


12% 


9% 


7% 


21% 



88 Because the sample sizes for media studies and communication; geography; ethnic, gender, and cultural studies; 
and writing are so small, results for those disciplines should be interpreted particularly cautiously. 
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e. Motivations for using online resources 

As illustrated in Table 2.11, instructors who use digital resources do so for a variety of reasons. 
The most frequently cited motivations were integrating primary source material into the course, 
improving students' learning, providing students a context for a topic, getting students excited 
about a topic, allowing faculty to do new things in the classroom, and obtaining resources not 
available at their institution. Other motivations included convenience and time savings (for 
themselves or their students), access to materials or teaching methods that would otherwise be 
unavailable, and pressure from students, colleagues, or administrators. 

Several faculty explained their reasoning: 

It makes my course fresh and allows me flexibility in my teaching and keeps me current 
on topics and resources. Students love it! 

— English instructor, Monterey Peninsula College 

Tire availability of primary sources has been crucial for the success of my teaching in 
history. Students have remarked what a difference it has made and I have noticed a big 
difference between this course with the availability of online primary sources to those I 
have taught before that were based on printed resources. 

— History instructor, UC Berkeley 

Classes held in immersive virtual reality projection settings such as 'portals' and 'caves' 
are especially useful for teaching students about architectural environments. 

— Architecture instructor, UCLA 

Teaching critical thinking was low on the list overall; this motivation was lowest at UCs (47%) 
and highest at community colleges (75%), as were the motivations to teach information literacy, 
to allow students more creativity, and to help faculty stay up to date with colleagues. 
Integrating research interests was relatively important across all institution types, perhaps as a 
corollary to the large number of faculty who maintain their own collections. 

Very few instructors felt pressured by their administration to use digital resources, and even 
fewer felt that it would help them get promoted or get tenure. Participants in the faculty 
discussion groups agreed that there were few rewards (in the form of job security, promotions 
and/ or status in their field) to be reaped from employing digital resources. 
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Table 2.11: Faculty survey: Motivations for using digital resources 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How much do you agree or disagree with the following 
statements about your reasons for using digital resources? I 
use digital resources in my teaching... 










Because it improves my students' learning. 


78% 


77% 


70% 


86% 


To integrate primary source material into the course. 


78% 


76% 


82% 


82% 


To provide students a context for a topic. 


75% 


72% 


77% 


81% 


To get students excited about a topic. 


73% 


69% 


75% 


84% 


Because it allows me to do things in the classroom that I could 
never do otherwise. 


68% 


67% 


69% 


68% 


Because it provides access to resources that we don't have at our 
college. 


61% 


51% 


80% 


76% 


Because it allows my students to be more creative. 


56% 


51% 


52% 


72% 


To teach critical thinking skills. 


56% 


47% 


61% 


75% 


To integrate my research interests into my course. 


55% 


53% 


60% 


59% 


Because I like or feel very comfortable with the new technologies. 


53% 


49% 


55% 


63% 


Because it is more convenient for my students and their 
schedules. 


51% 


50% 


49% 


55% 


To teach information literacy (i.e., evaluating the online materials 
themselves). 


43% 


36% 


54% 


56% 


Because it saves me time. 


41% 


41% 


40% 


41% 


Because my students expect or ask for more technology. 


38% 


35% 


38% 


43% 


Because it allows me to stay up to date with my colleagues. 


37% 


31% 


36% 


55% 


To provide students with both good and bad examples of 
different kinds of scholarship. 


32% 


25% 


35% 


49% 


Because it creates a sense of community for students enrolled in 
my course. 


32% 


29% 


29% 


40% 


Because I enjoy having my teaching practices and course 
materials available to anyone in the world who would like to use 
them. 


25% 


22% 


25% 


31% 


Because the administration encourages me to use digital 
resources more. 


23% 


18% 


28% 


32% 


To provide students a preview of the course before they register. 


22% 


21% 


13% 


27% 


Because it may help me get promoted or get tenure. 


9% 


8% 


11% 


12% 



f. Reasons for non-use 

We asked two sets of questions to assess reasons for non-use and barriers to use: one asked why 
they do not use resources, and the other asked specifically about barriers to use. 

Table 2.12 illustrates that the foremost reason faculty have for not using digital resources is that 
they simply do not support an instructor's current teaching approaches. Several instructors 
elaborated: 

Given a teaching style and materials that require one-on-one and/or group discussions 
regularly, . . . the computer is a poor substitute for being in a classroom where ideas bong 
off of each other and where we "talk” as people to people, where I can see body language, 
and where I can manage the flow. 

— Literature instructor, UCLA 
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I think there is a real danger of students' becoming too computer literate and "connected" 
in ways that undermine, or at least compete with, other crucial skills: argumentative 
writing, careful and critical reading of long texts, and oral argument. 

— Political science instructor, UCLA 

My courses do not lend themselves readily to digital resources. . . . Any hope of 
conversation with the students about the material disappears; class becomes another 
television show for them. 

— Foreign language and literature instructor, UC Davis 

There is evidence that PowerPoint and those other displays with bells and whistles etc. 
rot the mind. My students need to learn how to THINK and to READ BOOKS and, in 
the case of foreign languages, talk to real people. Their attention span is being annihilated 
enough with the huge number of "technical events" on television. 

— Foreign language and literature instructor, UC Davis 

J find digital technology inherently alienating and a distraction from the sense of human 
community and interpersonal communication I try to create. 

— Writing and art instructor, UC Santa Barbara 

Frankly, I just don't really want to use digital resources. Wltat's wrong with books 
anyway ? 

— History instructor, UC Berkeley 

A simple lack of time was a constraint on everyone, regardless of institution. 

I came across an adage that " email allows me to do in one hour what I never had to do 
before." So it goes with course WWW sites and digital instructional media too. 

— Art history instructor, UC Berkeley 

I am excited about the possibilities here to truly enhance teaching. For me the primary 
stumbling block is in having the time to explore and evaluate sources, not a lack of 
sources or a lack of belief that quality resources are out there. 

— English and writing instructor, Mendocino Community College 

I have not devoted enough time to finding out what is out there. I feel like I need a 
sabbatical just to learn to make efficient use of digital matter. 

— History instructor, Pomona College 

The reliability of digital content was a source of concern raised in the faculty discussion groups. 
Some participants felt a growing pressure to teach "web literacy" to undergraduates so students 
could better evaluate the credibility of digital resources. In addition to digital collections use, 
some faculty members had specific concerns about how students used search engines for 
course-related research, suggesting that web searching, in particular, ran the possibility of 
eroding "good" learning (e.g., search engine results decontextualize information by retrieving 
an orphaned page that lacks the context of the originating site). Two instructors expressed their 
concerns: 
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Plagiarism with online resources is a real problem. In addition, they all seem to believe 
that everything they find on the web is absolutely true, and they have no interest in, or 
ability to, determine the credibility of various sources found on the web. 

— Political science instructor, UC Santa Barbara 

Access to the web initially diminished my students' abilities as researchers - they 
substituted it for better print material - but this is slowly beginning to change. It 
remains an enormous issue, however in relation to plagiarism. 

— Architecture instructor, UC Berkeley 

Survey results, however, suggested that relatively fewer faculty have serious concerns about 
copying, plagiarism, and students' information literacy skills than we would have expected. 
Concerns about information literacy were somewhat more of a concern to community colleges 
than to UCs or liberal arts colleges, however (39% vs. 25%). 

Most of my students appear not to have learned how to do a good job of basic library 
research. I feel that should be a skill to be developed before others. 

— Anthropology instructor. East LA Community College 

I must teach students about using digital resources responsibly. 

— English instructor, Monterey Peninsula College 

Discussion groups and survey respondents also suggested that many faculty were jaded about 
keeping up with the "new new" thing. They were apprehensive about investing time in 
learning how to use new tools (i.e., they did not want to be beta testers) and felt that valuable 
time was wasted on technical development projects that had limited functionality and usability. 
Several elaborated: 

I once did a project on automating Dutch grammar drills and it turned into an 
ENORMOUS waste of time. It is hard enough to get my research done; I do not have the 
time to really work up new skills in this area. 

— Foreign language and literature instructor, UC 

Tlte technological environment has changed so rapidly on campus that it is very difficult 
to reuse materials without a great deal of very tedious reformatting. 

— Foreign language and literature instructor, UC Berkeley 
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Table 2.12: Faculty survey: Motivations for not using digital resources 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How much do you agree or disagree with the following 
statements about your reasons for not using digital resources? I 
don't use digital resources in certain teaching situations, 
because... 










They cannot substitute for the teaching approaches I use. 


75% 


78% 


81% 


66% 


I don't have time to use digital resources. 


66% 


67% 


72% 


61% 


Using them distracts from the core goals of my teaching. 


47% 


49% 


59% 


34% 


I don't want my students to copy or plagiarize material from the 
web. 


33% 


35% 


21% 


34% 


They are irrelevant to my field. 


30% 


30% 


38% 


28% 


Students don't have the information literacy skills to assess the 
credibility of digital resources. 


29% 


25% 


27% 


39% 


Digital material can be presented outside its original context. 


25% 


24% 


19% 


31% 



g. Barriers to faculty use of resources 

Faculty, including those active and enthusiastic in their use of digital resources, identified many 
obstacles to using these resources for teaching. They were unsatisfied both with their ability to 
find the resources they need and with the tools available to manage those digital resources in 
different contexts. One instructor felt particularly frustrated: 

As an art historian, I'd love to use more digital resources in teaching - especially as more 
and more interactive digital reconstructions of ancient and medieval monuments become 
available. However, I cannot afford to upgrade my computer and equipment on my own. 
Moreover, the VRC in my department is very stingy about letting faculty use the 
scanning equipment, which means that I haven't been able to do the kind of 
experimenting with digital images I'd like to be able to do. My answers on this survey 
will probably look very strange; it's because I am totally gung-ho about using digital 
resources but have not had the opportunity to use them in the way I'd like to! 

— Art history instructor. University of California 

The most-cited obstacles to the effective use of digital resources were the availability, reliability, 
and expense of the necessary equipment in the classroom. One faculty member explains 
his/her reluctance to use technology in the classroom: 

I find that the computer in class anchors me to a certain spot and at times to a certain 
order of presentation. I need freedom to improvise, change direction, and physically move 
around... Finally, I hate the tension that equipment introduces into the classroom, the 
fear of breakdown, the suspense, the frequent waste of time. . . . 

— English instructor, UC Santa Barbara 

Tire physical teaching facility is a big issue. I am currently carrying my laptop and 
projector from classroom to classroom and having to reconnect two or three times per 
teaching day. The rooms in which I teach have no online hook-ups, which is also a 
limitation. The physical burden of this technology can sometimes discourage me from 
using it. 

— History instructor, LA Harbor Community College 
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In one discussion group, a graduate student described a labyrinthine process for accessing 
classroom technology; she knew that her department owned the equipment she needed, but 
nobody could tell her where it was, who was responsible for it, or how to obtain access. 
Community college instructors reported the most trouble with access to technology, including 
both students' and instructors' computers and Internet access as well as institutional 
technology. Our faculty discussion groups suggested that in community colleges, the relative 
rank of a faculty member may be important. For instance, it was reported that part-time faculty 
were given low priority for using resources such as wired classrooms. 

In some cases, faculty were in a continual state of cobbling together internal and external funds 
to support innovative work (e.g., finding the funds for a systems administrator of new servers) 
because their institutions, although enthusiastic, could not provide the necessary resources. 

My department's budget model dates back to the mimeograph; thus I must acquire my 
own computers, film scanner, flatbed scanner, printer, and software. Some devices come 
from my extramurally-funded projects, but much is purchased out of pocket. 

— Art history instructor, UC Berkeley 

Often we have money to buy gadgets, but no money for training or maintenance. That's 
the biggest problem. 

— Literature and writing instructor, Porterville College 

Keeping equipment up to date is not taken seriously by those with the funds. 

— Anthropology instructor, UC Davis 

Other major obstacles included difficulty locating high-quality, pedagogically relevant materials 
from credible sources, and the sheer volume of available materials. Academic quality of 
materials was a concern for more UC (45%) and liberal arts faculty (40%) than community 
college faculty (24%). Forty- three percent of all faculty stated they did not have time to assess 
the credibility of available resources. 

Some faculty in specialized fields found too little material (or none at all) that met their needs or 
applied to their subject. One explained, "African materials are largely underrepresented in 
digital resources," and another said bluntly, "There are very few digital images available . . . for 
'nonwestern' fields in general." 

Even when the materials were available, organizing and archiving them remained a challenge. 
"Gathering sources and materials for my students and researching information to meet 
departmental needs are easy enough tasks but organizing it all is not," explained one instructor. 

Survey results suggest that copyright concerns are not a major barrier for most faculty and are 
cited by only one-third of respondents. 
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Table 2.13: Faculty survey: Barriers to digital resource use 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How much do you agree or disagree with the following 
statements? I have difficulty using digital resources the way I 
would like, because... 










I don't have reliable access to physical resources in my 
classroom(s). 


53% 


52% 


43% 


63% 


The digital resources are distributed in so many places that it is 
difficult for me to organize them for use in my teaching. 


45% 


43% 


56% 


46% 


There are too many resources out there for me to take advantage 
of —I am overwhelmed. 


44% 


42% 


51% 


47% 


I don't have time to assess the credibility of the available 
resources. 


43% 


42% 


48% 


44% 


The content I need or want is just not available online. 


41% 


47% 


48% 


25% 


The academic quality of available materials is too poor to meet my 
needs. 


39% 


45% 


40% 


24% 


I don't have reliable access to scanners. 


39% 


40% 


34% 


39% 


I don't know how to locate the online materials I need. 


36% 


33% 


43% 


40% 


I don't know how to save presentations to my computer so they 
can be run without a live connection. 


35% 


35% 


35% 


36% 


Available software is unsuitable for integrating audio or video into 
my course. 


34% 


31% 


40% 


40% 


Search engines provide irrelevant results for my needs. 


34% 


31% 


35% 


39% 


I have difficulty understanding the issues surrounding copyright 
and digital collections. 


33% 


33% 


33% 


34% 


My students don't have a high-speed connection. 


32% 


28% 


13% 


54% 


Course management software packages are inadequate for my 
needs. 


32% 


32% 


41% 


29% 


Websites I would use are unreliable, and I can't count on them 
being there when I need them. 


32% 


30% 


39% 


33% 


Available software is unsuitable for viewing and displaying digital 
images. 


31% 


29% 


39% 


31% 


My students don't have reliable access to computers. 


30% 


24% 


10% 


54% 


Web formats allow me to link to whole documents, but not to 
specific excerpts within a text. 


28% 


28% 


21% 


32% 


It is difficult to get server space or access to a server in order to 
store/ host digital resources for teaching. 


27% 


23% 


21% 


38% 


I don't have reliable access to a high-speed connection. 


21% 


19% 


15% 


32% 


I don't have reliable access to a computer. 


13% 


10% 


11% 


21% 



h. Activities for which support or assistance is important 

The most striking conclusion from the responses to this questions may be that support is 
needed for almost everything at relatively high levels (>70%), with little variation. The 
exception is evaluating the appropriateness and credibility of resources, which are relatively 
unimportant for UCs and liberal arts colleges (although they remain important to about two- 
thirds of community college faculty). 
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Table 2.14: Faculty survey: Support and assistance 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How important is it for you to have support or assistance with 
each of the following activities for your teaching? 










Obtaining or setting up technical infrastructure. 


82% 


81% 


89% 


84% 


Creating my own website. 


82% 


81% 


87% 


81% 


Digitizing existing resources. 


80% 


79% 


92% 


78% 


Learning how to use a learning management system. 


79% 


78% 


85% 


80% 


Importing resources into a course website or a database. 


79% 


79% 


81% 


77% 


Gathering, organizing, and maintaining digital materials. 


78% 


77% 


86% 


77% 


Integrating resources into a learning management system. 


78% 


75% 


83% 


81% 


Finding digital resources. 


72% 


69% 


75% 


78% 


Training students to find or evaluate digital resources. 


71% 


67% 


77% 


79% 


Interpreting copyright laws and/or securing copyright 
permission. 


65% 


62% 


69% 


70% 


Assessing the credibility of digital resources. 


50% 


43% 


50% 


69% 


Evaluating the appropriateness of resources for my teaching goals. 


42% 


35% 


38% 


60% 


How satisfied or dissatisfied are you with the support you have 
received from the following sources? 










Campus librarians 


84% 


85% 


83% 


84% 


Friends or family 


83% 


81% 


76% 


90% 


Graduate students 


80% 


81% 


64% 


75% 


Campus educational technology or IT support staff 


73% 


74% 


76% 


68% 


Other faculty 


71% 


70% 


72% 


74% 


Undergraduate students 


70% 


73% 


65% 


69% 


My departmental or college staff 


66% 


68% 


66% 


63% 


Workshops 


60% 


55% 


59% 


67% 


Online help or guides 


47% 


44% 


48% 


53% 



2. Personal collections 

The importance of personal collections was evident from faculty survey responses to the 
question: "Where do you find digital resources?" As shown in Table 2.8, the large majority of 
faculty (69%) reported that they use resources from their own collection, and those personal 
collections were the second-leading source of such resources (behind only search engines). 

In a separate question, we asked whether they (1) gather or maintain their own collection of 
digital resources, and (2) make their own digital resources available to others via the World- 
Wide Web. (See Table 2.15, below.) The responses were remarkably consistent. Nearly three- 
quarters (72%) said they maintain their own resources, while only 37% of those (or 27% of 
respondents overall) make their resources available to others online. 

The use of personal collections is fairly heavy across all disciplines (see Table H.2 in Appendix 
H). Use of personal collections as sources for digital materials is particularly common in 
anthropology/ archaeology (81%), art / architecture (81%), and history (74%); more than 58% of 
respondents in literature, writing, and political science say they rely on their own collections. 
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Table 2.15: Faculty survey: Personal collections 





All 


University 
of California 


Liberal arts 
colleges 


Community 

colleges 


N 


831 


522 


90 


206 


I gather or maintain my own collection of digital resources. 


72% 


71% 


71 %x 


75% 


(Of those who responded yes to the previous question) 

I make my own digital resources available to others via the 
World-Wide Web. 


37% 


37% 


38% 


37% 



In discussion groups and open ended survey responses, faculty discussed a variety of reasons 
for maintaining their personal collections; the most important was the need to annotate, 
manipulate, and organize the various resources and to present them in context within the 
course. Keeping their own copy of a digital item made them more confident that they'd be able 
to find the correct object again, and that it would still be accessible when needed. These 
personal collections resided largely on individual personal computers — rarely in any sort of 
database or on a server. Based on the free responses, it seems that faculty stored the resources 
in the format that was most convenient and that didn't require learning a new, more 
complicated system. 

It was not always easy to determine the original source of the items in these personal 
collections. Based on open ended survey responses and discussion group feedback, they appear 
to be a mixture of items collected from the Internet, those digitized from print sources 
(e.g., scanned from books), and those created by the individual faculty member (e.g., original 
documents written by the instructor or photographs taken by the instructor). Faculty tended to 
describe the whole collection as their own material, rather than focusing on the original sources 
of individual items. For faculty who rely heavily on primary sources, existing items culled from 
other sources may be the most valuable; even for these instructors, however, their individual 
annotations and organization may be the "added value" that makes the personal collection 
worthwhile. 

One faculty member described his reasons for creating his own digital resources: 

I personally create essentially all the digital materials that I use for my classes. I have not 
done much about integrating related materials created by others. I should probably do 
that, but time is short and I am lazy. It would probably take as much time to wade 
through others' work finding what is useful and how to adapt it than it is to make my 
own, which I know address my needs. 

— Linguistics and foreign language instructor, UCLA 

Faculty with substantial personal collections faced a variety of challenges in the online world. 
Some who had developed digital collections were frustrated by the demands of required 
collection maintenance and preservation. Often the resources (e.g., student assistance, funding, 
technical support staff) initially relied upon to build or digitize their collections had vanished 
when they were needed most. 

Several explained the challenges of integrating personal resources into new media, 

I own a personal collection of 40,000 35-mm slides, so to put it mildly, I am very invested 
in 20th century technology. I would need real help - both in machines and time - to 
convert teaching to PowerPoint, although I see some of its genuine advantages. 

— Architecture and geography instructor, UC Berkeley 
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I would really need a full-time visual resources person to convert, label, and store in an 
accessible way the thousands of slides in my own teaching collection. 

— Art history instructor, Whittier College 



3. Use by discipline 

Because of the complexity of the disciplinary analysis, we chose to summarize the results using 
principal component scores. As discussed above (page 4-16) principal component analysis is a 
method for simplifying a large number of quantitative variables into a smaller number of 
summary variables, based on patterns that emerge from the data themselves. Although some of 
the detail is lost by this simplification, the component scores provide a sense of respondents' 
characteristics on key dimensions. (See Figures 2.6-2.10, below; complete statistics can be found 
in Table H.2 in Appendix H.) Because the principal components are normali z ed to a mean of 
zero and a standard deviation of one, the scores do not tell us about individuals' absolute level 
of usage; rather, they measure usage relative to all survey respondents. (Negative scores 
indicate values lower than the survey average, and positive scores are greater than the survey 
average.) 

Although some of the specifics of this analysis may not be surprising, the wide range of 
differences is important. 89 

a. Writing 

Writing instructors are heavy users of general purpose and reference materials and of 
discussion and curricular materials, although they rarely use historical documents, maps, or 
primary source materials. They are more likely than average to use digital materials for online 
courses or student assignments, and less likely to present them in the classroom. As a 
discipline, writing instructors are the most interested in using digital resources to teach 
information literacy and critical thinking, and are the most heavily influenced by the 
expectations of their students and colleagues, although they also have a sense that the materials 
may not be appropriate for their discipline or teaching style. They are not overly concerned 
with whether materials are publicly and freely available. 

b. Literature and English language 

Instructors who teach literature and English tend to be near the survey average on most 
measures, with few extremes. They are slightly heavier users of general purpose and reference 
materials and lighter users of historical documents, maps, and primary sources and of data, 
news/ media, and governmental resources. They are also less likely than average to post digital 
materials online or to present them in class. They have some sense that digital materials are not 
appropriate for their subjects or teaching styles, although they do use them to teach information 
literacy and critical thinking more than the average instructor. Their only appreciable barrier to 
use is difficulty finding appropriate resources. 



89 Because the sample sizes for media studies and communication; geography; ethnic, gender, and cultural studies; 
and writing are so small, results for those disciplines should be interpreted particularly cautiously. 
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c. Foreign languages 

Foreign-language instructors are heavier-than-average users of images and audiovisual 
materials and light users of historical documents, maps, and primary sources, and of data, 
news/ media, and governmental resources. They tend to be motivated by pedagogical concerns 
when using digital resources, but they tend not to use them for teaching information literacy or 
critical thinking. These instructors deal with more barriers than average, including difficulty 
finding appropriate resources, a lack of high-end multimedia equipment and software, and a 
lack of equipment for their own use. 

d. Art and architecture 

Art and architecture instructors are much-heavier-than-average users of images and 
audiovisual materials, and light users of data, news / media, and governmental resources. They 
are more likely than average to assign digital resources to students for projects, portfolios, or 
studying, and to present the materials in class. These instructors agree that digital resources are 
appropriate for their teaching, and believe that they save time and improve convenience and 
access. 

e. Anthropology and archaeology 

Anthropology and archaeology instructors are heavier-than-average users of historical 
documents, maps, and primary source materials. They are more likely to post materials online, 
somewhat more likely to present them in class, and less likely to assign them for student 
projects. These instructors are more motivated than average by the desire to publicize their 
course materials, and they do not feel that digital resources are inappropriate for teaching. 

They are more troubled than the average instructor by the lack of appropriate, high-quality 
materials for their teaching. 

f. History 

Not surprisingly, history instructors are the heaviest users of historical documents, maps, and 
primary source materials; they also use more data, news/ media, and governmental resources 
than the survey average. They are more likely than average to present digital resources in the 
classroom. They are more motivated than average by pedagogical concerns, and less by their 
reputation and the expectations of others, or by concerns about time, convenience, and access. 
They feel somewhat more need for technical assistance than the average instructor. 

g. Political science 

Political science instructors use far more data, news/ media, and governmental resources than 
average and use fewer images and audiovisual materials. They tend to use digital materials for 
posting on the web, and are less likely to assign them for student projects, homework, or 
studying. They are not particularly motivated to use digital resources by the expectations of 
and pressure from students, colleagues ,or administrators, and they don't particularly need 
outside help with intellectual and content-based activities, such as finding digital materials and 
assessing their appropriateness and credibility. 
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h. Geography 

Although our sample of geography instructors was small (N=18), the results suggested a few 
interesting trends. Geography faculty were far-heavier-than-average users of data, 
news / media, and governmental resources, and were also heavier users of historical documents, 
maps, and primary source materials. They presented digital materials in the classroom much 
more than average. They felt strongly that digital materials were appropriate for teaching, 
although they were not particularly influenced by pressure from their students, colleagues, or 
administrators. They also needed less help than average with the technical tasks related to 
digital resource use. 

i. Ethnic, gender, and cultural studies 

Although the survey N was small (26), some trends were suggested for instructors of ethnic, 
gender, and cultural studies. They were heavier-than-usual users of data, news / media, and 
governmental resources and of historical documents, maps, and primary source materials, but 
lighter users of general purpose and reference materials. They were less likely to assign digital 
materials to their students. They were concerned about their students' misinterpretation of 
digital resources and lack of information literacy; they were not motivated by concerns for time, 
convenience and access, but were interested in publicizing information about their course or 
their work. These instructors had greater-than-average concerns about their own and their 
students' access to computers and high-speed connections; they needed more support than 
average with both technical and content-based activities. 

j. Media studies and communications 

Because the sample of instructors of media studies and communications was so small (N=13), 
caution is warranted in interpreting the results. These instructors were heavy users of images 
and audiovisual materials and of discussion and curricular resources, but relatively light users 
of historical documents, maps, and primary source materials; they use these materials in the 
context of online courses more often than average. These instructors do not find digital 
materials inappropriate for their teaching; they want to publicize their resources to others, but 
aren't particularly concerned about using materials that are already free or publicly available. 
They need less support than average with technical activities. 
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Figure 2.3: Faculty survey responses by discipline: What digital resources do you use? (part 1) 
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Figure 2.4: Faculty survey responses by discipline: What digital resources do you use? (part 2) 
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Figure 2.5: Faculty survey responses by discipline: How do you use digital resources in your 
teaching? 
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Figure 2.6: Faculty survey responses by discipline: What resources people use (principal 
components) 
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Figure 2.8: Faculty survey responses by discipline: Reasons for use and non-use (principal 
components) 
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Figure 2.9: Faculty survey responses by discipline: Barriers to use (principal components) 
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Figure 2.10: Faculty survey responses by discipline: Activities with which people need 
support (principal components) 
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4. The role of age in resource use 

Conventional wisdom suggests that digital resources are primarily the domain of younger 
instructors and that older faculty simply do not use them very much. Survey results support 
this theory to a limited extent: for almost all types of digital resources and all use scenarios, the 
oldest instructors (age sixty-two and up) were the lightest users (See Figure 2.11 and Table 2.16, 
below). We should note that although there seem to be real differences in level of digital 
resource use by age, as discussed below, an instructor's age alone is a weak predictor of digital 
resource use; other factors have a much greater effect on a person's use level (see "Regression 
and Path Analysis", page 4-55 below.) 

Beyond this statement, however, the results get more complicated. Many types of resources, 
such as news and media, digital film and video, digitized documents, data archives, and online 
journals are used more heavily by younger instructors. Other resources are much less 
influenced by age; images and visual materials and disciplinary portals, for example, are used 
heavily by instructors in all age groups (except the very oldest). Although very few instructors 
use personal online diaries ("blogs") in their teaching, the youngest instructors do so at a higher 
rate. 

As shown in Table 2.16, there are some differences in motivations for using digital resources 
among different age groups. Y ounger instructors are more likely to use digital resources for 
pedagogical reasons, believing that they will improve students' learning, help teach critical 
thinking skills or information literacy, and get students excited about a topic. Younger 
instructors also feel more comfortable with new technologies. Concerns about time, 
convenience, the availability of appropriate materials, and students' ability to deal with the 
materials, on the other hand, cut across the different age groups. 

It is also interesting that the oldest instructors reported the least need for support in almost all 
areas. It appears that they do not need help with the various digital activities because they are 
less interested in these activities, with or without support. When they did need support, 
though, the different age groups turned to different sources. Younger instructors were more 
likely to seek help from friends and family, workshops, or online sources, while older 
instructors were more likely to go to graduate students or campus librarians. 
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Figure 2.11: Faculty survey responses by age group: What digital resources do you use? 

% of respondents who use each resource (at least sometimes): 
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Table 2.16: Survey responses by age group (selected questions) 
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How often do you use the following types of digital resources in your 
undergraduate teaching: 
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To get students excited about a topic. 
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Because it provides access to resources that we don't have at our college. 
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To teach critical thinking skills. 


56% 


70% 


56% 
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Because I like or feel very comfortable with the new technologies. 


53% 


70% 


53% 


54% 


53% 


43% 


Because it is more convenient for my students and their schedules. 


51% 


60% 


51% 


51% 


53% 
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To teach information literacy (i.e., evaluating the online materials 
themselves). 


43% 


51% 


52% 


41% 


41% 


29% 


Because it saves me time. 


41% 


51% 


41% 


41% 


40% 


41% 


How much do you agree or disagree with the following statements 
about your reasons for not using digital resources? I don't use digital 
resources in certain teaching situations, because... 














I don't have time to use digital resources. 


66% 
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I don't want my students to copy or plagiarize material from the web. 


33% 
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Students don't have the information literacy skills to assess the 
credibility of digital resources. 
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How much do you agree or disagree with the following statements? I 
have difficulty using digital resources the way I would like, because... 
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I don't have time to assess the credibility of the available resources. 
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The content I need or want is just not available online. 
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The academic quality of available materials is too poor to meet my 
needs. 


39% 
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How important is it for you to have support or assistance with each of 
the following activities for your teaching? 














Obtaining or setting up technical infrastructure. 
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Creating my own website. 
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Digitizing existing resources. 
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Importing resources into a course website or a database. 
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Gathering, organizing, and maintaining digital materials. 
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Integrating resources into a learning management system. 
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How satisfied or dissatisfied are you with the support you have 
received from the following sources? 














Campus librarians 
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77% 


82% 


83% 


84% 


94% 


Friends or family 
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5. Heavy users, light users, and non-users 

Survey responses reveal a wide range of levels of use of digital resources, but the differences 
between light and heavy users are more complex than might initially be suspected. Figure 2.12 
and 2.13 show survey responses by the instructors' level of use of digital resources. (See also 
Table H.4 in Appendix H.) Using our composite index of site usage (see page 4-9, above), we 
divided respondents into four groups based on usage level: 

• Non-users to light users (usage index = 0-13, N=113), 

• Light to medium users (14-29, N=306), 

• Medium to heavy users (30-44, N=259), and 

• Heavy users (45-100, N=153). 

It is interesting to note that both light and heavy users face challenges and need support — but 
their needs differ. 

Light users complain primarily about their difficulty finding, managing, and assessing digital 
materials; they feel overwhelmed and unable to locate useful resources. Light users are also less 
likely to have reliable access to a computer or a high-speed connection. Medium to heavy users, 
on the other hand, are more likely to be concerned about their students' lack of computers or 
high-speed connections. The lack of high-end multimedia equipment and software is also more 
likely to be a barrier for medium to heavy users — things like server space, scanners, appropriate 
classroom technology, and more advanced, reliable software for integrating audio and video 
into their courses. 

Light users and non-users report that they need somewhat less support related to digital 
resources, presumably because they're not even attempting to use them to any significant 
degree. There is, however, a difference in the support needed by intermediate-level users and 
the heaviest users. The heaviest users need assistance obtaining and setting up technical 
infrastructure, creating websites, and dealing with copyright issues. Intermediate users are 
more likely to need help finding and managing digital content, including digitizing existing 
materials and importing them into databases or learning management systems. 

When they do need support, light and heavy users turn to different sources. Light users seek 
help from librarians, graduate students, and other faculty, but are unlikely to go to their 
campus' educational technology or IT staff. In addition to technology experts, heavy users also 
seek help from their family and friends and from online help or guides. 
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Figure 2.12: Faculty survey responses by use-level: Barriers and frustrations 
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I don't have time to assess the credibility of the available 
resources. 

The content I need or want is just not available online. 
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Search engines provide irrelevant results for my needs. 
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My students don't have a high-speed connection. 

Course management software packages are inadequate for 
my needs. 
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Available software is unsuitable for viewing and 
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My students don't have reliable access to computers. 

Web formats allow me to link to whole documents, but not 
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order to store/ host digital resources for teaching. 

I don't have reliable access to a high-speed connection. 

I don't have reliable access to a computer. 
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Figure 2.13: Faculty survey responses by use-level: Support and assistance 
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6. Highlights from the H-Net survey 

Overall, the results of the H-Net survey were strikingly similar to the California faculty survey. 
(See Figures 2.14 and 2.15, below, and Table H.7 in Appendix H.) The various types and 
sources of resources used were ranked in nearly the same order, as were methods and 
motivations for use. Where the two populations differed, the H-Net respondents seemed 
characteristic of heavier users, as might be expected among people recruited from an online 
community like H-Net. Also, the disproportionate number of history instructors among the 
respondents affected the H-Net results somewhat (the use of digitized documents and historical 
facsimiles was somewhat elevated, corresponding to the behavior of historians from the 
California faculty survey). H-Net respondents were also more likely to report hearing about 
digital resources from "professional societies or discussion lists," just as one would expect. 
Unfortunately, the low response from non-U.S. institutions made it difficult to examine the 
effects of national or cultural differences. In general, the correspondence between results of the 
two surveys reinforces the findings of the California faculty survey. 



http:/ / digitalresourcestudy.berkeley.edu 



4-52 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Figure 2.14: What digital resources do people use? (Faculty survey vs. H-Net survey) 



% of respondents who use each resource (at least sometimes): 



□ California faculty survey 

□ H-Net Survey 



Images or visual materials 

News or other media sources and archives 

"Portals" that provide links or URLs relevant to particular 
disciplinary topics 

Online reference resources 



Digital film or video 
Maps 

Online or digitized documents 
Audio materials 

Curricular materials and websites that are created by other 
faculty and/ or other institutions 

Digital readers or coursepacks 
Online class discussions 



Government documents 



Data archives 



Digital facsimiles of ancient or historical manuscripts 
Simulations or animations 



Personal online diaries 



0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
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Figure 2.15: Motivations for using digital resources (Faculty survey vs. H-Net survey) 



□ California faculty survey 

□ H-Net Survey 



because it improves my students' learning. 



to integrate primary source material into the course. 



to provide students a context for a topic. 



to get students excited about a topic. 

because it allows me to do things in the classroom that I 
could never do otherwise. 

because it provides access to resources that we don't have 
at our college. 

because it allows my students to be more creative. 



to teach critical thinking skills. 



to integrate my research interests into my course. 

because I like or feel very comfortable with the new 
technologies. 

because it is more convenient for my students and their 
schedules. 

to teach information literacy (i.e., evaluating the online 
materials themselves). 

because it saves me time. 



because my students expect or ask for more technology. 



because it allows me to stay up-to-date with my colleagues. 

to provide students with both good and bad examples of 
different kinds of scholarship. 

because it creates a sense of community for students 
enrolled in my course. 

because I enjoy having my teaching practices and course 
materials available to anyone in the world who would like 

because the administration encourages me to use digital 
resources more. 

to provide students a preview of the course before they 
register. 

because it may help me get promoted or get tenure. 



% of respondents who agree (strongly or somewhat): 

I use digital resources in my teaching... 

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 
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7. Regression and path analysis 

As the above results show, the use of digital resources is a complex phenomenon, influenced by 
a wide range of institutional, disciplinary, and individual factors. To gain a better 
understanding of digital resource use, we used regression and path analysis to explore the 
interrelationships between these multiple variables in predicting an instructor's level of use. 
Regression is a statistical technique for exploring the relationship between a set of predictor 
variables and a particular outcome, such as use of digital resources. 

Based on the literature and conventional wisdom, we identified twenty-one variables that might 
be expected to predict an instructor's digital resource use (see Table 2.17, below). These 
variables included characteristics of the individual respondent (age, gender, education, time 
since highest degree) and characteristics of respondent's teaching environment (institution type 
and discipline). Binary dummy variables were used for community college and UC teaching 
contexts, and liberal arts colleges were omitted, to avoid over-specifying the model. We also 
included respondents' level of enthusiasm about using digital resources and a measure of total 
use of computer technology, based on reports of the frequency with which respondents used 
computers, email, the worldwide web, and online library catalogs. We used our composite 
index of digital resource use as the dependent variable. (See page 4-9, above.) 

We performed a series of regressions, looking at each predictor's effect on digital resource use 
in isolation. Next we performed a multivariate regression, looking at the additive effect of all 
twenty-one variables, assuming each was acting independently and concurrent with all the 
others. The multiple linear regression model details the effects of each predictor variable on 
respondents' level of usage of digital resources, assuming a simple, linear relationship. 90 

The notion of simultaneous independent effect is strained when attempting to model what we 
know, from practical experience, to be a complex web of interactive causality. To address this 
issue, we created a somewhat more complex model by using path analysis to explore the total 
effect of each variable, including both direct and indirect effects. This approach provides a 
somewhat clearer picture of what is certainly a complex phenomenon. 

a. Regression and path analysis results 

Table 2.17 shows the results of individual regressions, looking at each predictor in isolation. 
Eleven of the twenty-one initial variables show statistically significant relationships with total 
usage. 91 Older instructors, for example, tend to use fewer digital resources or use them less 
frequently. It is important to note, however, that although this relationship is statistically 
significant, the effect size is very small: each additional year of age predicts a usage score that is 
0.23 points lower (out of 100). The total predictive power of this linear regression model 
(measured by the adjusted R 2 ) is also very weak: age alone accounts for only 2.5 percent of the 
total variation in use. 



90 It is important to keep in mind the combined assumptions of linearity and additivity when assessing the adequacy 
of the multiple linear regression model as a way of describing the relationships among the predictor variables and 
the distribution of actual usage as measured by the index. 

91 For this and all subsequent analyses, we examined statistical significance at the 0.05 level. Statistical significance in 
a linear regression model indicates that the regression coefficient differs from zero by an amount unlikely to have 
occurred by chance alone (due to the sampling process). 
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In fact, the linear model is not a very good one for any of the variables, as none has an adjusted 
R 2 of more than 10 percent, with the singular exception of respondents' enthusiasm about 
digital resources, which accounts for one-third of the total variation in our usage index. 

Overall, instructors who teach languages, architecture, geography, and history use more digital 
resources (or use them more frequently), while those who teach classics and those who teach at 
UC use fewer. Instructors who are more enthusiastic about digital resources are also heavier 
users, as are younger instructors, females, those who finished school more recently, and those 
who use computer technologies more frequently. 

It is worth noting that although these eleven variables are statistically significant, most of their 
effects are so small, and they account for such a small portion of the total variance in usage, 
they have essentially no predictive utility. Despite expectations to the contrary, a person's age, 
for example, provides almost no indication of whether the person will be a heavy user, light 
user, or non-user of digital resources. 

Table 2.17: Individual bivariate regressions: Predicting overall use of digital resources in 



teaching 



Variable 




Model adj. 
R 2 


Parameter 
estimate ((3) 


p-value 


Significant? 
(95% Cl) 


Age 


(Per year of age) 


0.025 


-0.23 


0.00 


* 


Years since degree 


(Per year) 


0.029 


-0.23 


0.00 


* 


Education level 


Four-point scale 


0.0005 


-1.25 


0.25 




Gender (female) 


Yes/no 


0.0045 


2.33 


0.04 


* 


Institution type 


Yes/no 


0.01 








UC 






-4.59 


0.01 


* 


Community college 






-1.76 


0.35 




Discipline 


Yes/no 


0.06 








English 






-2.32 


0.12 




Languages 






3.81 


0.01 


* 


Literature 






-2.64 


0.06 




Writing 






2.51 


0.10 




Anthropology 






2.61 


0.24 




Archaeology 






1.01 


0.76 




Architecture 






10.41 


0.00 


* 


Art history 






0.9 


0.70 




Art practice 






3.57 


0.09 




Classics 






-10.22 


0.01 


* 


Geography 






12.34 


0.00 


* 


History 






6.51 


0.00 


* 


Political science 






-0.31 


0.87 




Enthusiasm 


1-7 scale 


0.33 


4.92 


0.00 


* 


Total tech use 


0-16 scale 


0.07 


2.16 


0.00 


* 



Table 2.18 shows the results of the multivariate regression, using the same variables identified 
in Table 2.17. The total model has an R 2 of 0.39— a trivial improvement over the bivariate 
model containing only enthusiasm. Figure 2.16 offers a graphical representation of the 
statistically significant relationships; the line width is proportional to each predictor's 
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standardized partial regression coefficient (the parameter estimate, standardized by the 
variables' standard deviations). 

While eleven of these variables had a statistically significant bivariate correlation with the usage 
index, only seven do when all twenty-one predictors are taken into account simultaneously: 
teaching at UC; teaching architecture, classics, geography, or history; enthusiasm about digital 
resources; and greater use of computer technology. 

Other variables that were statistically significant when considered individually, such as age, 
have no significant direct effect. Thus age, for example, has only minimal direct impact on 
usage: although older people may use fewer digital resources, this can be best explained by 
their lower enthusiasm and lower exposure to computer technology. 

Table 2.18: Multiple regression: Predicting overall use of digital resources in teaching 



(Model adjusted R 2 = 0.39) 



Variable 




Parameter 
estimate (|3) 


p-value 


Significant? 
(95% Cl) 


Age 


(Per year of age) 


-0.05 


0.56 




Years since degree 


(Per year) 


-0.005 


0.95 




Education level 


Four-point scale 


0.17 


0.90 




Gender (female) 


Yes/no 


0.99 


0.32 




Institution type 


Yes/no 








UC 




-4.59 


0.0013 


* 


Community College 




-0.86 


0.64 




Discipline 


Yes/no 








English 




-0.87 


0.53 




Languages 




1.68 


0.22 




Literature 




-0.82 


0.51 




Writing 




1.33 


0.34 




Anthropology 




1.43 


0.46 




Archaeology 




-1.02 


0.72 




Architecture 




6.70 


0.02 


* 


Art history 




0.64 


0.77 




Art practice 




2.78 


0.19 




Classics 




-7.07 


0.02 


* 


Geography 




10.53 


<0.0001 


* 


History 




5.75 


<0.0001 


* 


Political science 




0.38 


0.82 




Enthusiasm about digital 
resources in teaching 


1-7 scale 


4.47 


<0.0001 


* 


Total technology Use 


0-20 scale 


1.03 


0.001 


* 



We can improve this model by using path analysis, a technique that examines both the indirect 
and direct effects of a set of variables. Enthusiasm, for example, is a major predictor of digital 
resource use, while several other variables are statistically significant predictors of enthusiasm: 
technology use, teaching in a community college setting, teaching English, or teaching art 
practice. Figure 2.17 shows a path analysis diagram for this model. All predictive relationships 
shown in this model are significant at the 95 percent confidence level; arrows for non-significant 
relationships are not shown. For each relationship, the standardized partial regression 
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coefficient is noted next to the arrow. (Negative coefficients indicate negative relationships; for 
example, teaching English predicts a lower enthusiasm about digital resource use.) 

For each variable, the paths trace both direct and indirect effects on a respondent's total usage 
of digital resources. We can see, for example, that while education level has no direct effect on 
digital resource usage, it has indirect effects via technology usage and via disciplines such as art 
practice. While art practice has an impact only via enthusiasm, technology use exerts both a 
direct and an indirect effect on digital resource usage. 

b. Example: The effect of computer technology use 

The combined effects of technology use are highlighted in Figure 2.18. The direct effect of a 
respondent's level of technology use on digital resource use has a standardized partial 
regression coefficient of 0.12. This indicates that an increase of one standard deviation in 
technology use tends to result in an increase of 0.12 of a standard deviation of digital resource 
use. 

Technology use also exerts an indirect effect on digital resource use by predicting a person's 
enthusiasm level. The total coefficient of this indirect relationship can be calculated by 
multiplying the partial coefficients along the predictive path, resulting in a standardized 
regression coefficient of 0.25 x 0.51 = 0.13 for the indirect relationship. Combining the indirect 
and direct relationships, the total effect of technology use on digital resource use has a 
coefficient of 0.12 + 0.13 = 0.25. Thus the total impact (direct plus indirect effects) of technology 
use, according to this model, results in a predicted 0.25 standard deviation increase in digital 
resource use for an increase of one standard deviation in technology use. This is twice the 
magnitude of the effect estimated when considering only the direct effect. 

c. Summary 

This analysis reinforces the importance of a person's individual attitudes about digital 
resources. It seems obvious, but the instructors who use more digital resources in teaching are, 
by and large, those who are most enthusiastic about it. There are a few small exceptions to this 
rule: to a small degree, instructors in geography and history use more digital resources than 
one might expect based on their enthusiasm levels alone, while those who teach at UC use 
fewer. Perhaps in these disciplines, established work practices and scholarly expectations may 
reduce the importance of individual enthusiasm. Institutional characteristics of UC, on the 
other hand, may inhibit some faculty from using digital resources the way they would like. 

Why people feel positively about digital resources is itself a complicated phenomenon. 
Instructors from community colleges and from art practice tend to report higher levels of 
enthusiasm, as do those who are heavier users of computers in their day-to-day non-teaching 
activities. These findings may underscore a substantial difference in mindset between the 
community colleges and other institution types. Perhaps community college faculty, faced with 
less-well-prepared students, are excited by the prospect of using digital resources as a "hook" 
for student engagement; perhaps they are generally more open to pedagogical change or 
experimentation. 

English instructors, on the other hand, tend to be less enthusiastic about the role of digital 
resources in their teaching, perhaps because of their reliance on texts, or perhaps because of 
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their sense that digital resources cannot be easily integrated into their teaching style. Even 
among English instructors, however, there is a range of attitudes, and those who are more 
enthusiastic about digital resources use them more. 

Age, education level, and gender seem to have no significant impact on a person's enthusiasm 
about digital resources or on their level of use of those resources. 

The path analysis diagram (Figure 2.17) highlights the importance of individual opinions; it also 
lays out the complicated set of relationships between those opinions, an instructor's 
demographic characteristics, and the institutional and disciplinary settings in which they teach. 

Figure 2.16: Multiple regression: Predicting overall use of digital resources in teaching 
(Model adjusted R 2 = 0.39) 




- 0.15 



Total use of 
digital 
resources in 
teaching 
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Figure 2.17: Path analysis: Predicting overall use of digital resources in teaching 



Institution type: 
UC 




http:/ / digitalresourcestudy.berkeley.edu 



4-60 



Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Figure 2.18: Path Analysis: Predicting overall use of digital resources in teaching 
(Effects of technology use highlighted) 
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8. Clustering and user profiling 

a. Background 

Cluster analysis is a method for assigning objects to groups suggested by the data rather than 
by a priori assumptions. Objects that cluster together are more similar to each other than to 
objects in a different cluster. 

We employed cluster analysis in an attempt to simplify the organization of our data about 
faculty users and non-users of digital resources into meaningful and descriptive categories. We 
identified a preliminary and limited cast of user and non-user types from the academic 
community based on their patterns of use of digital resources as well as their motivations and 
frustrations. We also hoped that cluster analysis might uncover any hidden or latent clusters of 
faculty that were not immediately obvious from the summary data due to the complexity of 
multiple variables. 

b. Methods 

We generated data-based clusters, based on a large number of variables simultaneously. After 
assigning each observation to a cluster, we described the general characteristics shared by the 
members of each cluster. 

A variety of clustering techniques are available, which may produce many different sets of 
clusters from the same observations. In addition, we could have assigned clusters based on 
many different combinations of variables; the number of clusters defined is also somewhat 
arbitrary; we could have defined more or fewer. These alternate clustering methods could have 
produced results that, though vastly different, would still be equally valid. 

We created disjoint clusters of faculty survey respondents, with each respondent as a member 
of only one cluster, using hierarchical "tree clustering" based on Ward's minimum variance 
method (Ward 1963). We defined the clusters based on instructors' individual attitudes and 
behaviors and not on their institutional, disciplinary, or demographic characteristics; we 
defined ten clusters. 

We generated clusters based on the following variables. (See page 4-16 for a more detailed 
description of the principal components.) We normalized variable values before clustering, to 
eliminate the effect of varying scales. 

1) Total amount of digital resource use 

2) Enthusiasm about digital resources 

3) Use of personal collections - 

4) What Resources People Use (5 principal component scores) 

5) How People Use Digital Resources (4 principal component scores) 

6) Reasons For Using or Not Using Digital Resources (8 principal component scores) 

7) Barriers to Use (2 principal component scores) (We omitted barriers related to the availability 

of instructors' equipment, students' equipment, and high-end multimedia equipment, since 
they seem to be more related to the instructor's institutional setting than to individual 
characteristics.) 

8) Activities With Which People Need Support (2 principal component scores) 
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We also experimented with several alternate criteria for clustering; we developed clusters based 
on institutional and disciplinary affiliation, individual demographics such as age, gender, and 
education level, amount and type of digital resource use, and other behavioral and attitudinal 
measures, to see whether these different clustering criteria produced similar clusters. 

When disciplines or institution types group together in this clustering model, it suggests that 
they really do have similar needs, attitudes, or behaviors. Note that hierarchical clustering 
methods are not designed to provide any statistical significance measures (such as p-values). 
(For a more thorough introduction to cluster analysis, see Everitt, 1980). 

c. Results 

Our ten clusters comprised a total of 608 survey respondents. (The remaining 223 survey 
respondents could not be included in the model due to missing values.) A complete statistical 
summary of the ten clusters can be found in Appendix K. 

The resultant clusters still display a great deal of intra-cluster variance. In addition, very few 
variables for each cluster are far from the population mean. Table 2.19 lists the characteristics of 
each cluster that are more than one standard deviation from the population mean; only the one 
characteristic that is marked with an asterisk is more than two standard deviations from the 
mean. 
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Table 2.19: Cluster analysis highlights 



Cluster 


Demographics, 
institution, and discipline 


Overall use and 
enthusiasm 


Principal components: what & how; personal 
collection use 


Principal components: 
reasons, barriers, and support 


1 

(N=60) 




Low total 
use; low 
enthusiasm 


Low use of general-purpose/ reference 
resources, of images/ audiovisual 
resources, and of historical/ primary- 
source materials. Low use for student 
projects, in-class, and web posting. 
Low use of personal collections 


Unlikely to believe to 
believe that digital 
resources improve 
student learning. 


2 

(N=106) 










3A 

(N=48) 


Unlikely to be from 
community colleges 


Low total 
use 


Low use of images and audiovisual 
material; low in-class use 




3B 

(N=95) 








High need for technical 
support 


3C 

(N=63) 






High use for web posting 




4A 

(N=62) 






High use of historical and primary- 
source materials; high use for in-class 
presentation 


Above-average belief 
that that digital resources 
improve student 
learning; below-average 
concern for time, 
convenience and access 


4B 

(N=36) 


High proportion of 
people with 
bachelor's degrees 
or less. 




Low concern about making 
information freely available 


Few problems finding 
digital resources; low 
need for intellectual 
support 


4C 

(N=68) 




High total 
use; high 
enthusiasm 


High use of images and audiovisual 
materials; high use in class and for 
student projects; high use of personal 
collections 




5A 

(N=48) 


Predominantly 

female 




High use of discussion and curricular 
materials; high use for online courses. 




5B 

(N=22) 


High proportion 
from community 
colleges; low 
proportion from 
UC. 


High total 
use; high 
enthusiasm 


High use of images and audiovisual 
materials and of discussion and 
curricular materials; high use for web 
posting and for online courses*; high 
use of personal collections 


Above-average belief 
that digital resources 
improve student 
learning; below-average 
concern that digital 
resources are not 
appropriate for their 
teaching. 



* Only the item marked with an asterisk is more than two standard deviations from the population mean. 



d. Discussion 



The clusters presented above may provide an alternative way to parse the survey responses. By 
identifying particular user groups according to multiple dimensions, we can ask. What are a 
particular group's needs and concerns? What group does a particular service or system best 
assist? How can services be tailored to the needs of particular user types? 
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This array of clusters is still fairly complex; because of the high degree of intra-cluster variation 
and scarcity of extreme values, it is difficult to pinpoint or summarize most of the clusters. As 
an example. Figure 2.19 displays a scatterplot of two of the principal component scores used in 
clustering: the use of general-purpose and reference materials and the use of images and 
audiovisual materials. Different clusters are marked by different colors and shapes of data 
points. 



Figure 2.19: Scatterplot: Use of general-purpose and reference materials vs. use of images 
and audiovisual materials 
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Overall, the groups seem not to be very tightly clustered. It appears that our survey population 
has few natural or latent clusters, and does not naturally divide into clear and distinct "types". 
We observed a wide range of faculty characteristics, and these characteristics occurred in a wide 
variety of combinations. People in the same institution or discipline may approach digital 
resources with very different perspectives; those who use the same types of resources may do 
so in different ways or for divergent reasons; and those who are driven by similar motivations 
may use the resources in different ways and experience different barriers and frustrations. 

When we attempted to define clusters using alternate sets of criteria, we also found groups that 
were only loosely clustered. Slightly different methods and criteria produced largely different 
groupings, with only a weak connection between the clusters defined here and those defined 
using different sets of criteria. This variation again suggests the lack of strong natural clusters 
in this sample. 
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In our main model, we found only a few clusters with any notable correlation to institutions or 
disciplines; this suggests that these particular patterns of behaviors and attitudes are more 
common among certain institutional or disciplinary environments. Most of the behavioral and 
attitudinal patterns, however, seem to be found in a wide variety of institutions and disciplines. 

e. User profiling 

One possible use for a cluster analysis like that presented here would be to develop user 
profiles or personas, which could then be used as a tool for digital resource planning or design. 

By way of background, user profiling defines and categorizes a set of generalized user types 
(including non-users), as a method for characterizing the target population of a given digital 
resource or set of digital resources. Types may be derived from theoretical or policy goals that 
identify dimensions of interest, such as institution type, organizational context, disciplinary 
focus, or age of respondent. Types may also be derived from the statistical grouping of 
responses into clusters of relative similarity with respect to individual attributes, access 
requirements, information needs, tasks, and working environments. User profiling can enable 
developers to create resources that meet the needs of a target audience and to isolate 
representative user types for conferring and testing during stages of development. With the 
ubiquity of digital development projects, user profiling of both potential users and non-users 
may become a standard practice for describing use . 92 

Based on the cluster statistics above, one could describe a semi-fictional composite person who 
is typical of each cluster. Alternately, one could select an actual person, near the center point of 
each cluster, on which to base the persona. If one is planning to develop user profiles based on 
actual survey data, cluster analysis can help ensure that the profiles are well distributed across 
the range of actual user characteristics. 

9. Summary of findings: Faculty survey 

The complexity of the faculty survey results points to the many interacting variables that impact 
use. These variables include institution type, discipline, age, and faculty idiosyncrasy. In 
general, however, we can say that digital resources vary widely in type, purpose, and perceived 
value. Instructors use a dizzying range of objects, including personal collections and a mix of 
free and commercial resources, for a wide range of educational purposes and goals. 

We see a broad spectrum of user types, ranging from the non-user, to the inexperienced, novice 
user, to the highly proficient and advanced user of digital resources. Non-users are themselves 
diverse. Often pejoratively referred to as "Luddites" by those more enamored of technology, 
non-users in fact include those who are passionately opposed to the use of technologies in their 
classroom for a variety of valid pedagogical reasons (e.g., they cannot substitute for preferred 
teaching approaches). Non-users also include self-described enthusiasts frustrated by technical 
and non-technical barriers, and those simply without enough time to think about, let alone use, 
technology in teaching. 



92 In Grant's study of digital cultural content evaluation publications, for example, she reports that 78 percent of her 
sample employed user profiling as a method for summarizing audience characteristics of a given resource (Alice 
Grant Consulting, 2003). Bischoff and Allen (2004) note that museums, because of their need to understand tend to 
break down their "visitors" into several categories when compared to libraries. 
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Highlights of our analyses are discussed below. 

WItat people use and where they find resources. Our findings underscore the broad array of digital 
resources available to and used by faculty. Clearly, no one site has a monopoly on the 
educational resource domain. We were struck by the frequent use of sites not specifically 
designed for educational purposes (including a wide variety of news, current events, 
governmental, and reference resources). Faculty in our sample, like students, say they most 
commonly use Google-type search engines to find resources, including images. Faculty's 
personal collections were the second most common source of resources. Online journals and 
public/ free image databases are also high on the list of preferred ways of locating desired 
resources. 

Curricular materials. In light of the significant activity around the open courseware movement, 
we were particularly interested in what faculty said about the use of online curricular materials. 
Among institutions, the community colleges were slightly heavier users than other institutions 
(although overall use of curricular material relative to other resources was low). Among the 
various disciplines, foreign language, writing, art and architecture, and geography faculty are 
the heaviest users of curricular materials, and anthropology, literature and language, history, 
and political science faculty are the lowest. These findings suggest that the former disciplines 
may be more amenable to adopting digital curricular materials, and the latter more interested in 
customized products. 

How resources are used. Faculty responses to questions about how resources are used suggest 
that resources are predominantly used in classroom presentations and assigned to students for 
review and research projects. Very few instructors reported using digital materials for online 
courses or discussions, although the former practice is more common at community colleges, 
and more common in a few specific disciplines (e.g., writing). 

Motivations for using online resources. The most frequently cited motivations were integrating 
primary source material into the course, improving students' learning, providing students a 
context for a topic, getting students excited about a topic, allowing faculty to do new things in 
the classroom, and obtaining resources not available at their institution. Teaching critical 
thinking was lower on the list overall but highest at community colleges, as was the motivation 
to teach information literacy. Integrating faculty research interests was relatively important 
across all institution types, perhaps as a corollary to the large number of faculty who maintain 
their own collections. 

Very few instructors felt pressured by their administration to use digital resources, and even 
fewer felt that it would help them get promoted or get tenure. Participants in the faculty 
discussion groups agreed that there were few rewards (in the form of job security, promotions 
and/ or status in their field) to be reaped from employing digital resources. 

Reasons for non use. The foremost reason faculty have for not using digital resources is that the 
materials simply do not support an instructor's current teaching approaches. This is an 
important message to those who develop tools and content and those who provide campus 
infrastructure to faculty. It also points to the importance of assessing faculty needs, goals, and 
pedagogies first, and then adapting technology to fit those needs. 
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A simple lack of time was a constraint on everyone, regardless of institution. Fewer faculty 
have serious concerns about copying (33%), plagiarism, and students' information literacy skills 
than we would have expected from talking with librarians and other administrators who saw 
these as perhaps the biggest hurdles for faculty. Information literacy was somewhat more of a 
concern to community colleges (39%) than to UCs or liberal arts colleges, however (27% and 
25% respectively). Discussion groups and survey respondents also suggested that many faculty 
were jaded about keeping up with fast changing technologies. 

Barriers to faculty use of resources. Overall, we can say that it is not at all easy for most of our 
respondents to use the mass of digital resources available to them. Faculty, including those 
active and enthusiastic in their use of digital resources, identified many obstacles to using these 
resources for teaching. They were unsatisfied both with their ability to find the resources they 
need and with the tools available to manage those digital resources in different contexts. The 
most-cited obstacles to the effective use of digital resources were the availability, reliability, and 
expense of the necessary equipment in the classroom. Community college instructors reported 
that both students and instructors had trouble with access to computers, Internet connections, 
and other technology. 

Other major obstacles included difficulty locating high-quality, pedagogically-relevant 
materials from credible sources, and the sheer volume of available materials. Even when the 
materials were available, organizing and archiving them remained a challenge. In some cases, 
faculty were in a continual state of cobbling together internal and external funds to support 
innovative work (e.g., finding the funds for a systems administrator of new servers) because 
their institutions, although enthusiastic, could not provide the necessary resources. 

Survey results suggest that copyright concerns are not a major barrier for most faculty— 
perhaps because they have given up trying to understand or comply with the law, and are 
going about their business the best they can. In our discussion groups, nuances of copyright 
and fair use eluded many faculty, irrespective of institution type; few faculty members had a 
clear understanding of how copyright laws applied to their use of digital collections in the 
classroom, despite voicing concern. Individuals (confidentially) admitted to using material and 
ignoring copyright issues, due to time pressure and convenience. Because of new laws and 
ongoing legal issues (e.g., downloading music from the web) many faculty looked to their 
administration for clearer guidance about their legal rights and an easy way to obtain copyright 
permission, although few campuses had clear policies in place. 

Activities for which support or assistance is important. The most striking conclusion from the 
responses to this question may be that support is needed for almost everything at relatively 
high levels with little variation. The exception is evaluating the appropriateness and credibility 
of resources, which are relatively unimportant for UCs and liberal arts colleges (although they 
remain important to about two-thirds of community college faculty). The finding for UCs is 
somewhat unexpected, as we have heard from administrators that this is among the most 
important areas for which faculty need support. 

Personal collections. Nearly three-quarters of faculty reported that they gather or maintain their 
own collection of digital resources. Those personal collections were the second-leading source 
of such resources (behind only search engines). We suspect in many cases that scholarly 
practice is linked tightly to pedagogical approach in various sectors of humanities and social 
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science teaching, leading many faculty to place a high value on integrating their own resources 
and research into their teaching practice. 

As Borgman (2003) has suggested, the ability to create personal digital libraries, which allow the 
integration of resources from diverse sources for reuse, is a need that most users will have at 
one time or another. This finding corresponds with Penn State's Visual Image User Study 
(Pisciotta et al., 2002), the work of UC's California Digital Library (Farley, 2004), and ARTstor's 
research (Roger Schonfeld, personal communication), all of which found extensive use of 
personal digital collections among their institutions' faculty as well as a desire to have tools that 
allow easy integration of those resources with other software and systems. 

Beagrie (2005) sees the use of personal collections increasing and discusses the challenges of 
managing these growing collections. A variety of tools are being developed to assist faculty 
with organizing and maintaining their collections, and institutions are experimenting with ways 
to assist in this endeavor. Additionally, a number of institutions are beginning to grapple with 
how their faculty can share their personal collections with colleagues (Farley, 2004), perhaps by 
reaggregating these collections into institutional repositories. Our work suggests that the 
feasibility of such sharing is an open question (Harley et al, 2003). Although many faculty 
members in our survey expressed an interest in sharing their materials with others, somewhat 
fewer seemed interested in using others' collections. 93 (On the H-Net survey, where we asked 
more detailed questions, more than three-quarters of instructors used their own personal digital 
collections while fewer than one-third used a colleague's.) 

The centrality of personal collections represents a significant challenge in the design of online 
resources. For example, many online databases, repositories, and content-management systems 
assume that users will store or manipulate digital resources online within the context of that 
particular system; our research suggests that in fact this is not how faculty typically work. That 
is, faculty have an aversion to learning new systems, and identify "easy" as the most important 
quality in a digital resource or tool. Proprietary systems that keep the content "locked up" in 
various ways for intellectual property reasons also pose a barrier to faculty work practices, 
making it difficult for users to reaggregate, manipulate, and organize the materials. Difficulty 
tracking down the original source of these personal materials, and ascertaining their copyright 
status, is a nontrivial challenge for institutions. 

Disciplinary differences : The variation in responses among disciplines demonstrates again that the 
humanities and social sciences are not a monolith regarding how different resources are valued 
and what motivates faculty to use them. The findings suggest some caution is warranted when 
generalizing about the importance of any one type of resource for the higher education 
community. For example, faculty who use texts extensively may depend on different kinds of 
sources, for different pedagogical goals, than those in more traditionally "image-centric" 
disciplines. Political scientists, contrary to the aggregate, are not heavy users of images. Maps, 
historical documents, and primary source materials are heavily used by historians, 
anthropologists and archaeologists, as well as by geographers, but less so by writing instructors 
and those in English and foreign languages. 

As reinforced in our discussion groups, those in art and architecture appear to have different 
profiles than their colleagues in other disciplines, including a tendency to use resources for 



93 Approximately 72 percent of faculty said they maintained a personal collection but only 27 percent said they make 
their material available to others online. 
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student assignments and as a way to save time. Those who teach subjects that require three- 
dimensional visualization and / or historical reconstruction may need particularly sophisticated 
resources and tools, and the support that such applications demand. 

Even instructors who use similar resources may use them in very different ways and have 
different needs and concerns. Use of images is widespread across a variety of disciplines; 
however there remains variation in the types and quality of images needed, how they are used, 
and an instructor's specific needs in selecting them. The result is that an image database 
designed to serve art history instructors may not meet the needs of historians or language 
instructors. The latter have lower expectations regarding resolution and detail, and may not 
want to be bothered with high levels of complexity that often accompany sophisticated image 
databases. 

Heavy users, light users, and non-users. When analyzed by level of use, our data suggest that both 
light and heavy users face challenges and need support— but their needs differ. The difference 
in barriers and support among light and heavy users' needs suggests a model of "sequential 
hurdles." Advanced or heavy users are not as bothered by the issues that trouble beginners 
(finding, organizing, and assessing content), but not because these issues don't apply to them; 
rather, the advanced users have already successfully dealt with these issues and surmounted 
these barriers to reach their current level of usage. Only after they've solved these more basic 
problems do the more advanced issues become relevant. 

Finally, the survey data suggest that what people (university administrators or technical 
support staff) think they know about faculty will depend on whom they talk to. Needs, 
opinions, and behavior vary in crucial ways by age, discipline, and institution; the needs 
expressed by one group of faculty are unlikely to match those from a different discipline, age 
group, or experience level. Such variation in user needs may make one-size-fits-all programs or 
resources unlikely to serve the wide range of potential users. If institutional educational- 
technology or library-support programs are geared to either the "mean" or the most extreme 
outliers (or the most vocal constituencies), it is likely that many users and potential users will be 
dissatisfied or underserved. 
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GOAL 2B: TRANSACTION LOG ANALYSIS AND WEBSITE SURVEYS 

Principal authors: Jonathan Henke, Ian Miller, Charis Kaskiris, David Nasatir, Ph.D., 
Diane Harley, Ph.D. 

A. Introduction 

This analysis focuses on evaluating methods for tracking actual site usage. Specifically, we 
conducted a pilot test on two local sites, using a combination of server transaction log analysis 
(TLA) and online surveys to determine if and how these methods can elucidate patterns of use 
of digital resources. Our charge was not to generalize our findings about users of targeted sites. 
We only analyzed the efficacy and efficiency of TLA and online site surveys as methods for 
understanding use generally. 

Transaction log analysis is a widely used (or at least widely discussed) method for studying the 
usage of particular websites. It is a particularly valuable method because the usage data are 
collected automatically and passively; the method records actual behavior rather than relying 
on self-reports. Many sites also use web-based surveys, either alone or in combination with 
transaction log analysis, to learn more about their users. Surveys can be used to create a richer 
profile of the site's visitors and their attitudes, behavior, and motivations. 

Although these two methods are widely used, there seems to be little consensus about the best 
way to implement them and to report the results (Troll Covey, 2002; Mento and Rapple, 2003). 
This lack of consensus makes it difficult to interpret statistics for different sites and to compare 
one site with another. Both TLA and online surveys can be time-consuming and labor-intensive 
and unless research and analysis methods are sound, the results may be ambiguous or even 
misleading. Our goal is to explore methods that are effective, efficient, and yield high-quality, 
reliable results that assist site owners in practical decision making. 

Because TLA and online surveys explore slightly different aspects of a site's use and users, they 
can be complementary tools, and the combination of the two may allow a deeper understanding 
of a site's use than either alone. In this study, we set out to explore the benefits and challenges 
of these two user research methods by pilot testing their efficacy on two specific websites. 

By way of providing context, we first provide an overview of these two methods, including 
their key challenges and limitations, such as the inability of TLA to identify individual users, 
and the difficulty in obtaining adequate response rates to online surveys. Some challenges, 
such as respect for users' privacy and confidentiality, are a particular concern for educational 
and non-profit organizations. 

We then describe a pilot implementation of TLA and online surveys in combination on two 
local sites, and the results of that pilot test. From that pilot test, we draw conclusions about the 
utility of these two methods, and which particular analysitic methods may provide the most 
valuable and efficient results. 

1. Studying the users of specific online resources 

As discussed above, studying a website's users can provide information to help a content 
provider improve an online service or create new resources to meet emerging needs. Interviews 
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with site owners suggest a significant (and often unsatisfied) desire for a richer and more 
complete understanding of the site's users and potential users. This knowledge can be a useful 
tool for a site owner or developer's strategic decision-making, by helping them make optimal 
use of scarce resources. 

Studying a particular site's users may provide insight into the following: 

• User demographics 

• User needs and motivations 

• User preferences 

• Resource utilization levels within a particular timeframe 

• Changes in usage over time 

• Details of user interaction with existing materials 

• Levels of repeat use 

• Users' assessment of site usefulness. 

B. Background 
1. Transaction log analysis 

Transaction log analysis (TLA) takes advantage of the computerized log files that automatically 
record online access to any website (see sidebar). By analyzing these logs, one can determine a 
number of characteristics of the site's users and summarize total site use. TLA is rare among 
research methods in that it allows the researcher to measure the actual online behavior of a 
site's users, rather than relying solely on self-reports. 

Our research has employed available expertise and related projects to explore a set of practical 
methods for TLA that are compatible with different types of library and non-library collections. 
According to a recent CLIR publication, there are significant challenges to assessing the use and 
usability of digital collections through transaction log analysis (Troll Covey, 2002). 

• Because the logs identify only the client computer, it is usually not possible to identify 
individual users or track them over time. It can be difficult to determine which log 
records are associated with the same user. The user's IP address is often used as a proxy 
for a user identifier, but the IP address is not a perfect identifier in all cases: 

o The same user may visit a site from several IP addresses. People with dial-up 
connections or on other types of networks will have dynamically assigned IP 
addresses that vary from session to session, or even within a session. 
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o Several users may share one IP 

address. A public library or campus 
terminal may be used by many users 
to access a site. Also, small networks 
frequently share one IP address, so 
different users on different computers 
may still appear to originate from a 
single IP address. 

• Each IP address can be associated with a 
particular hostname, but the IP address 
(or even the hostname) may not reveal 
anything of interest about the actual 
person. 

• Analyses may attempt to use hostnames 
to identify characteristics of individual 
users, such as their country of origin, 
educational status, or institutional 
affiliation. However, these analyses can 
be unreliable or even misleading: 

o Researchers may attempt to identify 
users from colleges and universities 
by looking for hits from .edu domains. 

However, many educational users rely 
on commercial dialup for home access, 
where the commercial IP address has 
no bearing on the user's educational 
status. 

o Hostnames can be used to attempt to 
locate users geographically, 
particularly for hits from international 
(country code) domains. However, 
most users still originate from generic 
top-level domains (without country 
codes), which are difficult or 
impossible to pinpoint geographically 
based on the hostname or IP address 
alone. 

o In addition, hostname lookup may not 
be 100 percent reliable, due to 
incomplete or out-of-date DNS 
records; these records may be less 
reliable for international domains. 

• Other, more advanced geolocation techniques exist and have improved dramatically 
since the 1990s, but they can be quite expensive and are still not 100 percent accurate. 
MIT's OpenCourseWare analysis used the Akamai service to help locate its users 
geographically (Carson, 2004). 



What is a web transaction log? 

Web transaction logs (or access logs) are 
recorded automatically by web server software 
every time a file is requested. A separate entry is 
made for every file request, including HTML 
pages, images, style sheets, script files, PDF 
documents, and other types of files. Each log 
entry will typically include the following items: 

• The IP address or hostname of the computer 
requesting the file (the user’s computer) 

• The username, if authentication is required 
(if the user had to enter a login and password 
to access the file) 

• The date and time of the request 

• The method (GET or POST) 

• The filename requested 

• The version of HTTP used by the client’s 
browser 

• HTTP response code (indicating whether the 
file was successfully retrieved, and if not, 
what error message was returned) 

• The number of bytes transferred 

• The referrer (the URL from which the file 
was requested; e.g., the page from which the 
user clicked on a link to reach the requested 
page) 

• The client’s user agent (which identifies the 
user’s browser and operating system and also 
identifies if the file was requested by a robot 
or automatic web crawler) 

• Cookies included with the request, if 
applicable. 

A web server will also generally be configured to 
create an error log, which records every time an 
error message is returned; other software, such as 
Internet search engines, may have their own 
specialized logging system (e.g., recording which 
searches were performed). We analyzed only 
standard user access web logs, because they are 
so consistent from site to site, and because they 
contain important information about site usage. 



http:/ / digitalresourcestudy.berkeley.edu 



5-3 





Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Proxy servers limit the reliability of server 
logs: if the requested page is in the proxy 
server's cache, the web server will not be 
contacted, and will have no record of that 
access. 

Not all web browser events are logged by 
the web server. For example, the web 
server is generally oblivious to the user 
pressing the back button, because the 
page will be reloaded from the web 
browser's cache. Even though hours may 
pass, when a user re-visits a site later, that 
site may still be loaded from the browser 
cache instead of the website, effectively 
evading logging. Other events such as 
scrolling the window, switching 
applications, and periods of computer 
inactivity are also not logged. 

Cookies can serve as a better user 
identifier than an IP address, but while 
every Internet user has a relatively 
inflexible IP address, users can control the 
cookies you place on their system (and 
many users block cookies). For example, 
a cookie can be copied to other computers, 
deleted, or systematically modified by 
users for their own purposes, all of which 
will impact logging. 

More advanced TLA techniques (using 
some high-end analysis tools) may require 
extensive site modifications. Many 
commercial packages require the 
placement of a special HTML tag on each 
page to facilitate the software's best 
features. Other techniques require 
JavaScript code embedded in the site's pages or invisible Macromedia Llash files which 
set Llash "cookies." These modifications require considerable expertise and place an 
additional burden on the site designer or manager, particularly for small or understaffed 
organizations. 



Sample web server transaction log 

169.229.132.204 - - [ 10/Feb/2004 : 12 : 20 : 4 8 - 
0800] GET / HTTP/1.1 304 - - Mozilla/5.0 

(Windows; U; Windows NT 5.1; en-US; rv:1.0.2) 
Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 10 - 
0800] GET / HTTP/1.1 200 4991 - 

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 15 - 
0800] GET /site_styles . css HTTP/1.1 200 3318 
http : //digitalresourcestudy .berkeley . edu/ 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 15 - 
0800] GET /title-text-only .gif HTTP/1.1 200 
2586 http : //digitalresourcestudy .berkeley . edu/ 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 16 - 
0800] GET /title-bg2_rl_cl . gif HTTP/1.1 200 
6534 http : //digitalresourcestudy .berkeley . edu/ 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 16 - 
0800] GET /title-bg2_rl_c2 . gif HTTP/1.1 200 
25119 

http : //digitalresourcestudy .berkeley . edu/ 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [ 10/Feb/2004 : 12 : 32 : 26 - 
0800] GET /index. html HTTP/1.1 200 4991 
http : //digitalresourcestudy .berkeley . edu/ 
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; 
rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 28 - 
0800] GET /about. html HTTP/1.1 200 6051 
http : //digitalresourcestudy .berkeley . edu/index . 
html Mozilla/5.0 (Windows; U; Windows NT 5.1; 
en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02 

169.229.132.204 - - [10/Feb/2004 : 12 : 32 : 29 - 
0800] GET /docs. html HTTP/1.1 200 3172 
http : //digitalresourcestudy .berkeley . edu/about . 
html Mozilla/5.0 (Windows; U; Windows NT 5.1; 
en-US; rv:1.0.2) Gecko/20030208 Netscape/7.02 



(NCSA Combined Log Format, generated by the 
Apache 2.0.x web server. Other log formats 
exist, including the W3C Extended Log Format 
used my Microsoft IIS. The two formats contain 
similar information, formatted differently.) 



Despite all of these challenges and limitations, transaction log analysis still has two major 
advantages over most other user research methods. First, it captures the actual behavior of real 
users in their own real-use environments; it does not rely on biased self-reports or artificial, 
laboratory-based use scenarios. Second, because TLA records behavior passively without 
requiring users' active participation, it can capture a much broader spectrum of uses and users 
than can surveys, focus groups, or other methods. These advantages make TLA a worthwhile 
user research methodology, in spite of the challenges. 
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2. Privacy and confidentiality 

Issues relating to privacy and confidentiality can be a concern when performing TLA; these 
concerns may be better defined at educational or non-profit institutions than in the private 
sector. Colleges and universities have traditionally had much higher standards for protecting 
individual privacy than have commercial institutions. Libraries in particular tend to have strict 
policies protecting their patrons' privacy, both on and offline. While commercial websites may 
be specifically designed to track individual browsing behavior (by requiring registration and 
login, or by using cookies), academic websites will rarely use these same techniques. Even 
anonymous log data may be more tightly controlled because of privacy concerns. 

For a variety of reasons (including unease with provisions of the USA PATRIOT Act that 
particularly target libraries), such institutions often have well-developed policies for protecting 
(or destroying) any identifying information about their patrons. Unfortunately, there is very 
little clarity about how these policies apply to online transactions or to user logs. At the most 
basic level, it is unclear whether transaction logs (and IP addresses in particular) should be 
considered "identifying information." (An IP address is not necessarily associated with an 
individual; even when it is, it is rarely possible to determine who the person is.) At some 
institutions, transaction log data can be used for internal analyses but not shared with external 
research partners. 

3. Online surveys 

In the past decade, web-based surveys have become more widespread, for a variety of reasons 
(Fricker and Schonlau, 2002; Gunn, 2002). Online surveys provide some cost and convenience 
advantages over other survey modes, but they also raise some problems that warrant careful 
consideration. 

Online surveys can take a variety of forms. Surveys can be administered online as part of a 
traditional, well-developed survey methodology involving a defined population of interest, an 
explicit sampling method for generating a representative sample, a well-thought-out 
recruitment strategy, carefully calculated response rates, and statistical estimates of the 
likelihood of response bias. 

Increasingly, however, online surveys are posted on a website and made available to anyone 
who happens upon them. These surveys rarely have a defined population or sampling method; 
with no way of tracking those who do or don't complete the survey, it is often impossible to 
report a response rate or estimate response bias. 

When one designs a survey instrument for online administration, a variety of new options are 
available for question structure, layout, and design (Gunn, 2002; Schonlau et al., 2002; Faas, 
2004). Important issues in instrument design include question wording, survey navigation and 
flow, skip patterns, survey length, and the graphical layout of the instrument. 

Computerization allows the design of more complicated skip patterns and question 
randomization. Additionally, it is possible to program automatic data checks and verification 
to disallow the entry of inconsistent responses. 

The automation of data collection and analysis can result in an economy of scale, making web- 
based surveys much more cost efficient, especially for large sample sizes. Automation can also 



http:/ / digitalresourcestudy.berkeley.edu 



5-5 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



mean that data (and basic analyses) are available in a much shorter timeframe— even 
instantaneously. 

An online survey can be a valuable complement to transaction log analysis for studying the use 
and users of a website; while TLA can reveal users' actual online behavior and usage patterns, 
surveys can reveal users' motivations, goals, attitudes, and satisfaction levels. 

A more detailed exploration of techniques for survey design, administration, and analysis can 
be found in Rossi, Wright, and Anderson (1983) and Fowler (2002). 

a. Survey response rates 

Survey response rates are an important concern, as rates for all types of surveys have been on 
the decline since the 1990s (Johnson and Owens, 2003; Baruch, 1999). Evidence suggests that 
response rates for online surveys are lower than for other media and continue to shrink (Fricker 
and Schonlau, 2002). 

Measuring response rates is a particular challenge for web-based surveys, partly because of the 
tricky definition of "response." Bosnjak and Tuten (2001) identify seven distinct response types, 
including lurkers (who view a survey without responding), drop-outs (who complete the 
beginning of a survey without continuing), item non-responders (who omit individual 
questions), and complete non-responders. 

To boost response rates, various rewards can be offered to increase participant motivation. The 
use of rewards and incentives can introduce response bias, however, as the people who are 
motivated to respond by a specific reward may not be representative of the whole study 
population. 

In traditional social science survey research, sampling methods are designed to ensure that the 
survey respondents are representative of the population of interest. If the sample is 
representative and the response rate is high, the survey results can shed light on the 
characteristics of the population. If, on the other hand, response rates are low or the sample is 
known to be non-representative, it is possible — even likely — that the survey results will be 
misleading. (Note that a large response rate alone is no guarantee that the respondents are 
representative.) 

However, sampling techniques and the measurement of response rates are a particular 
challenge when a survey is posted online and made available to any web user anonymously, 
without active recruitment or sampling. In such an environment, the population of users and 
the characteristics of the respondents are essentially unknown, making it difficult to report 
response rates and even more difficult to estimate the survey's response bias. The lack of 
knowledge of the complete population also makes it difficult to design appropriate sampling 
frames. 

Most basic statistics texts and guides to survey practice devote considerable space to a 
discussion of sampling. Minimizing sampling bias requires the use of proper sampling 
techniques — decisions made by the researcher about the subset of the population to whom the 
survey will be administered. Discussions of sampling techniques and analyses typically assume 
that the sample will be a small subset of the population and that the survey response rate will 
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be quite high — two assumptions that are becoming less justified. In the world of online 
surveys, it may be economically feasible to offer the survey to most or all of the population of 
interest. (In our pilot tests, we presented the surveys to 100 percent of the relevant 
populations.) At the same time, response rates within that sample are often vanishingly small. 

In this increasingly common scenario, sampling bias (the effect of the researchers' decisions 
about survey targets) has become less important, and non-response bias (the effects of 
individuals' decisions about whether to respond to the survey) is paramount. Unfortunately, 
the statistical literature (at least that accessible to the non-specialist) has provided very little 
guidance about how to deal with these challenges. Even the language and terminology of 
sampling can be unhelpful; the term "sample" sometimes refers to the people who are 
presented with the survey and sometimes to those who actually respond. In online settings, 
low response rates and self-selection bias may be unavoidable, since researchers may have no 
way to follow up with prospective subjects and no way to encourage their participation. 

Below, we propose a methodology for assessing response rates and response bias using the 
combination of online surveys and analysis of server transaction logs. (See page 5-20.) 

C. Pilot testing 

We conducted a pilot test on two local sites, using a combination of TLA and online surveys to 
determine if and how these methods can elucidate patterns of use among different types of 
institutions. 

Through the pilot tests, we explored the ease and efficiency of implementing these two methods 
on specific sites; we also attempted to identify the specific questions, analyses, and techniques 
that would provide the most useful information for understanding and comparing the usage of 
the sites. 

Given the enormous amounts of data that can be gleaned from transaction logs, we limited the 
range of possible TLA data to collect. We focused on a few key usage measures, including 
overall volume of site usage, domain name/IP address of users, user geography (based on 
hostnames), analysis of originating pages, and tracking of visitor persistence based on IP 
addresses. We believe that these measures may provide a relatively robust picture of actual 
collection usage using a straightforward, repeatable methodology. 

We selected two sites for our analysis: 

SPIRO 94 : The SPIRO website provides online access to the UC Berkeley Architecture 
Department slide library and comprises over 63,000 images (about 20% of the library's physical 
holdings). Images are cataloged by historical period, geographic location, object name, personal 
name, and subject using a variety of controlled vocabularies, including the Getty Trust's Art 
and Architecture Thesaurus. (The site includes lists of the metadata used for cataloging.) 
SPIRO's web interface supports advanced searching on any or all of these access points, but it 
does not support easy browsing of individual database items. (A side effect of the absence of a 
browsing interface is that the individual images and their metadata are not indexed by the 
major search engines.) This interface seems well suited to reference librarians and others who 



94 http:/ /www. mip.berkeley.edu/spiro 
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know precisely the data they seek; it may be less helpful for a person who is less familiar with 
the subject area or with the collection. 

The Jack London Collection 95 : The Jack London Collection features a wide variety of resources 
about the early-twentieth-century American author. The site includes the full text of nearly all 
of London's novels, short stories, and other writings; 113 images; a variety of biographical 
materials; facsimiles of his original correspondence; a range of curricular materials for teachers 
and students, mostly at the high school level; and links to related sites with further information. 
The collection can be accessed by browsing (and therefore can be indexed by search engines) as 
well as by searching. The user interface seems accessible and understandable to both novices 
and experts and seems designed with high school and college students in mind. 

We placed short surveys on the homepages of both sites and analyzed the survey results in 
combination with the transaction logs from the same period. The survey instrument can be 
found in Appendix N. 

D. Software comparison and selection 

One major goal of this project was to evaluate the appropriateness of different software tools for 
performing TLA on digital resource collections. In addition to examining existing software 
options, we examined whether it would be more efficient and effective to develop our own 
analysis software tools and methods. 

Our requirements were that the system be able to (1) handle the large volume of log data 
generated by our targeted digital collections, and (2) provide the detail and flexibility required 
by our analysis plan (below). 

A number of software packages exist for performing TLA. We evaluated several free and 
commercial packages for use on our pilot tests and for their suitability for TLA in other 
situations. We compared six widely used tools: three free packages (AWStats , Analog, and 
Webalizer) and three commercial packages (Wusage, Urchin, and WebTrends) that range from 
inexpensive to high-end enterprise-level systems. Details about the various features of the 
different packages are available in Appendix M. In general, greater power and flexibility come 
at the cost of increased complexity, decreased ease-of-use, and higher price. 

Because our methodology entailed experimenting with a variety of non-standard analyses — 
including the combination of survey and log data— we were unable to find any packages with 
the power and flexibility that we required. Many commercial analysis packages assume an 
established methodology and are not as flexible and customizable as we required. 

Instead, we developed our own customizable transaction-log-analysis package. We used a 
series of Perl scripts to clean and transform the raw logs; the logs were then imported into a 
normalized database in a standard RDBMS system; a series of database queries were then used 
to produce the analyses of interest. We also evaluated the strengths and weaknesses of this 
custom-developed system, compared with the more widely available options. 



95 http:// sunsite.berkeley.edu/London 
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E. Data processing 

We performed initial data processing to prepare the transaction logs for analysis. First, we 
performed DNS reverse resolution to convert each logged IP address (e.g., "169.229.132.204") to 
a hostname (e.g., "cssell.berkeley.edu"). For a variety of reasons, some IP addresses could not 
be resolved to a hostname (perhaps due to a misconfiguration of the user's ISP); these IP 
addresses were still uniquely identified, but we were unable to analyze their hostname 
information. From the pilot sites' logs, about one-fourth of IP addresses were unresolved. 

We then identified browsing sessions, which we defined as a series of page requests from the 
same IP address and user agent, without thirty consecutive minutes of inactivity. 96 For analysis 
purposes, we divided the files accessed into "pages" (HTML files) and "non-pages" (images, 
stylesheets, scripts, PDF files, etc.). We retained records for unsuccessful requests (errors such 
as "page not found"), but did not include these in our analyses. We also excluded from our 
analysis hits from known "spiders" or "robots" (automated programs which "crawl" the web to 
create search engine indexes). 

For each of the pilot sites, we combined the site transaction logs with the survey response 
database, linking each survey response with the respondent's session in the transaction log. 
Finally, we converted the log and survey data into a relational database (PostgreSQL), which we 
accessed with a statistical analysis package (SAS) to perform our analyses. Table 3.1 shows the 
file size and processing time required for each pilot site. 

For one of our pilot test sites, the data processing procedure was complicated by the 
organization's privacy and confidentiality concerns. In response to these concerns, we 
developed a method for "anonymizing" the transaction logs; this "anonymization" may have 
had a small effect on the reliability of the results. 97 



96 The combination of IP address and user agent is able to disambiguate some situations where multiple users share a 
single IP address. 

97 Privacy and confidentiality issues had a direct effect on our pilot tests. Reluctance to share raw user logs for fear of 
identifying individual users and the evolving and fluid nature of specific policies for such sharing created 
considerable challenges for us as third-party researchers. We settled on a compromise methodology for one site: we 
developed a method for "anonymizing" the user logs by partially randomizing the IP address. The modified logs 
allowed us to identify each user's Class C address (the first three numbers of the IP address) but not the exact 
computer. The pilot site's manager performed the "anonymization" before providing us with the logs. We believe 
that in most cases this didn't affect our identification of the user's domain. 

This "anonymization" process had several implications on our TLA results for this site. The "anonymization" 
probably decreased the accuracy of the IP-address-to-hostname conversion, resulting in some misidentified 
hostnames. To estimate the number of misidentified hostnames, we ran the anonymization process twice more to see 
how often the results agreed. Based on this simulation, we estimate that anonymization resulted in an incorrect 
hostname approximately 6 percent of the time. 

A mid-study policy shift interfered with matching survey responses to log data. We were forced to use timestamps 
to reconstruct the mapping. The global DNS database, which we relied on for hostname resolution, is a bit of a 
"moving target," as IP addresses and hostnames are changed over time. Because of the lag time as a solution 
developed, the DNS database had experienced several months of changes; as a result, we had more unresolved 
addresses (and presumably more misidentifications) than we would have if we had performed the lookup more 
promptly. To estimate the speed at which DNS changes, we resolved the same IP addresses again approximately six 
months later; we found that for both sites the number of IP addresses which could not be resolved to a hostname 
increased by 20 percent over the time period. Many of these issues would have been moot if we had been analyzing 
our own data, rather than obtaining logs from a partner organization. 
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Table 3.1: TLA data processing summary 





SPIRO 


Jack London 


Raw log file size 


214 MB 


315 MB 


Number of lines 


885,173 


1,425,273 


Running time 98 


~1 hour 


~3 hours 



F. Pilot test results 

In this section, we present the combined results of the online surveys and the transaction log 
analysis conducted on both pilot test sites. For each major set of research questions, we detail 
the specific metrics and analysis strategies that seem to provide the best insight into the sites' 
users and usage. 

Table 3.2 summarizes the analysis of the two sets of transaction logs. 



98 Postgres 7.3.x on lxl.8GHz P4 Xeon, 1GB RAM, 10K RPM SCSI storage. 
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Table 3.2: Transaction log analysis results summary 





SPIRO 


Jack London 


Logging period 


April 1, 2004 - 
May 31, 2004 
(61 days) 


March 27, 2004 - 
May 28, 2004 
(63 days) 


Number of sessions 


54,375 


145,956 


Number of unique IP addresses 


38,962 


97,284 


Number of sessions per IP address 
(mean ± SD) 


1.4 ±4.45 


1.5 ± 2.71 


Session length: 






Minimum 


0 sec 


0 sec 


Qi 


0 sec 


0 sec 


Median 


1 sec 


2 sec 


Q3 


30 sec 


113 sec 


Maximum 


4,837 sec 


5,320 sec 


Session entry-points: 






Home page 


30% 


23% 


Internal page ("deep linking") 


70% 


77% 


User persistence (by session): 






First-time users 


89% 


83% 


Repeat users 


11% 


17% 


User hostnames: 






.com 


18% 


19% 


.net 


25% 


24% 


.org 


1% 


1% 


.edu 


9% 


2% 


International TLDs 


22% 


11% 


U nknown/ unresolved 


25% 


43% 


Referrers 






Search engines 


26% 


31% 


berkeley.edu pages 


30% 


29% 


Other .edu pages 


2% 


5% 


Other 


25% 


10% 


Not provided 


18% 


24% 
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Table 3.3: Survey responses 





SPIRO 


Jack London 


Which title best describes you? 


N=106 


N=433 


University/ college instructor/ professor 


26 (25%) 


32 (7%) 


Undergraduate student 


14 (13%) 


76 (18%) 


Graduate student 


15 (14%) 


31 (7%) 


Independent researcher/scholar 


11 (10%) 


49(11%) 


Librarian 


15 (14%) 


10 (2%) 


K-12 instructor 


5 (5%) 


22 (5%) 


K-12 student 


5 (5%) 


154 (36%) 


Other 


15 (14%) 


59 (14%) 


For what purposes do you use the Jack London site? 
(Check all that apply.) 99 


N=58 


N=235 


Conducting research 


19 (33%) 


113 (48%) 


Creating presentations, including lectures 


10 (17%) 


31 (13%) 


Developing teaching materials 


15 (26%) 


23 (10%) 


Making my own collection of digital resources 


6 (10%) 


15 (6%) 


Other 


8 (14%) 


53 (23%) 


How often do you use the Jack London site? 


N=45 


N=196 


Today is my first time 


30 (67%) 


145 (74%) 


Daily 


2 (4%) 


5 (3%) 


Weekly 


6 (13%) 


5 (3%) 


Monthly 


3 (7%) 


15 (8%) 


Every six months 


2 (4%) 


7 (4%) 


Once a year 


0 (0%) 


1 (1%) 


Less than once a year 


1 (2%) 


3 (2%) 


Other 


1 (2%) 


15 (8%) 


Where are you accessing the site from now? 


N=45 


N=192 


Home 


21 (47%) 


121 (63%) 


Dormitory 


0 (0%) 


5 (3%) 


On campus office/ lab 


7 (16%) 


17 (9%) 


Library 


6 (13%) 


12 (6%) 


Office/ lab 


7 (16%) 


21 (11%) 


Other 


4 (9%) 


16 (8%) 



99 Percentages may not total 100% due to multiple selections; respondents were allowed to check multiple 
checkboxes. 
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SPIRO 


Jack London 


How often do you do the following activities on the Jack 
London site? (check all that apply ) 100 


N=47 


N=203 


Find text 


7 (15%) 


89 (44%) 


Download text 


3 (6%) 


53 (26%) 


Find images 


19 (40%) 


55 (27%) 


Download images 101 


20 (43%) 


34 (17%) 


Find images to license 


3 (6%) 


19 (9%) 


Browse to see what's available 


18 (38%) 


85 (42%) 


Use the site as a reference tool to verify citations 


10 (21%) 


58 (29%) 


Other 


2 (4%) 


12 (6%) 


What is your affiliation? 


N=40 


N=167 


Research university 


25 (63%) 


11 (7%) 


4-year college 


4 (10%) 


25 (15%) 


Community college 


0 (0%) 


6 (4%) 


High School 


1 (3%) 


49 (29%) 


Elementary or middle school 


4 (10%) 


30 (18%) 


Library 


2 (5%) 


2(1%) 


Museum 


0 (0%) 


0 (0%) 


Governmental agency 


0 (0%) 


6 (4%) 


Business 


0 (0%) 


6 (4%) 


Other 


4 (10%) 


32 (19%) 



1. Who are the users? 

Tables 3.2 and 3.3 give some indication of who is using each site. Survey respondents from the 
Jack London site were primarily K-12 students and teachers; those from SPIRO were primarily 
affiliated with colleges and universities. 

The survey responses also suggest that the overwhelming majority of site users consists of 
irregular or occasional users rather than regular users. This corresponds with the TLA results 
from Table 3.2, which indicate that the majority of users are first-time rather that repeat visitors 
(at least during the logging time period). 

2. Usage over time 

Figures 3.1 and 3.2 show site traffic over time for both pilot sites. Note the dips in usage during 
week 1 (which corresponded to spring break for UC Berkeley and many other colleges) and 
during week 3 (spring break for many local primary and secondary schools). The two sites 
follow similar trends, but Jack London usage is more affected by the high school spring break. 
Figure 3.3 shows files per session for each week of the study, which may give us a sense of the 
intensity or qualitative type of use of each site over time. Files-per-session holds fairly constant 



100 Percentages may not total 100% due to multiple selections; respondents were allowed to check multiple 
checkboxes. 

101 This question was made more specific for SPIRO to distinguish between thumbnails and full-size images. The 
frequency reported for SPIRO has been aggregated. 
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for the Jack London site; at SPIRO, on the other hand, sessions during weeks 7 and 8 included a 
notably greater number of files. This time period roughly corresponds to UC Berkeley's final 
exams. The parallels between SPIRO's usage and UC Berkeley's academic calendar suggest that 
Berkeley coursework accounts for a large portion of SPIRO's usage. The elevated number of 
files per session during final exams (Figure 3.3) suggests that usage during this period is not just 
greater, it is also qualitatively different. The typical usage session during exams is more 
intensive, involving more files and covering a broader range of material. 



Figure 3.1: Jack London traffic (site visitors per week) 




Week 
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Figure 3.2: SPIRO traffic 




Week 
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Figure 3.3: Files per session (comparison) 




Week 



3. User domains and referrers 

Figures 4 and 5 show the top-level domains of users' hostnames, with .edu domains broken 
down by individual institution. We observe that SPIRO has a much higher portion of use from 
.edu domains, which corresponds to the survey results. SPIRO also has a higher percentage of 
its use from international domains. 

Note that we cannot conclude from this chart the actual fraction of users who are from colleges 
and universities, since many college and university users access the site via non-.edu hosts. 
(Most home access is from a private ISP.) However, the difference between the two sites is 
suggestive, especially in combination with the survey responses. 

Figures 6 and 7 show a similar breakdown of the domain of referring sites — the pages from 
which people clicked on a link to reach each of the pilot sites. The sites show similar patterns, 
with 7 percent of users reaching the site from an international site and 32 percent from another 
.edu site (most often another berkeley.edu site). Thirty-one percent of Jack London traffic and 
26 percent of SPIRO traffic was referred by search engines. 
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Figure 3.4: Domains of users. Jack London 




Figure 3.5: Domains of users, SPIRO 
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Figure 3.6: Referrers, Jack London 




br 

1% 



Figure 3.7: Referrers, SPIRO 




4. Search engine queries 

Tables 3.4 and 3.5, below, list the specific search engine queries that led people to each pilot site. 
Note that the queries which led people to the Jack London site include a variety of specific 
pieces of content (specific literary works, "Jack London biography," etc.) that probably directed 
people to specific internal pages. The top queries that directed people to the SPIRO site, on the 
other hand, tended to be general terms that referred to the whole site. This is undoubtedly 
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because the individual items within the SPIRO collection are not indexed by the major search 
engines. It is unclear from this analysis if people are simply not searching for things like "Frank 
Lloyd Wright building photographs," or if (more likely) those searches are not directing people 
to SPIRO's materials. 

It is also interesting that several search terms that rank highly on SPIRO are probably "false 
positives" — cases where SPIRO is probably not what they were looking for. For example, 219 
people found SPIRO by searching for "group names," a search which led them to a page from 
SPIRO's controlled vocabulary listing (a list of the metadata used to catalog the collection). It is 
unclear exactly what these searchers were hoping to find, but it was probably not the controlled 
vocabulary page. 

Table 3.4: SPIRO: Search terms used to locate site 



Search Term 


Frequency 


spiro 


339 


group names 


219 


architecture 


184 


slide 


177 


architectural images 


155 


California architecture 


54 


database architecture 


47 


architecture image 


32 


architecture images 


30 


berkeley architecture 


29 


great architecture 


24 


slide library 


23 


image database 


23 


image library 


22 


architecture database 


21 


image id 


20 


[unknown Japanese characters] 


18 


photos of architecture 


14 


P 


13 


[unknown Japanese characters] 


13 
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Table 3.5: Jack London: Search terms used to locate site 



Search Term 


Frequency 


jack london 


11,177 


call of the wild 


2,749 


the call of the wild 


2,703 


white fang 


1,401 


to build a fire 


573 


martin eden 


568 


jack 


508 


jack london biography 


356 


the sea wolf 


266 


sea wolf 


237 


london 


205 


the call of the wild by jack london 


200 


call of the wild by jack london 


194 


london images 


167 


jack london books 


158 


john barleycorn 


150 


biography of jack london 


145 


jack london short stories 


143 


jack london call of the wild 


126 


moon face 


116 



G. Online survey representativeness 

We obtain a richer picture of site usage by analyzing surveys and transaction logs concurrently 
for the same site during the same time period. However, a new level of analyses is made 
possible by actually merging the two datasets, person-by-person. In particular, combining 
online surveys with transaction log analysis of the same site during the same time period allows 
new techniques for measuring the survey's response rate and for estimating response bias. The 
transaction logs allow us to measure the full population of site users during the study period — 
every user who viewed the site's homepage and therefore had the opportunity to take the 
survey. The transaction logs also allow us to describe everyone in the target population 
according to a few behavioral measures, based on their actual browsing patterns on the site. 
(Additional analyses would be required to see if site usage during the study period was typical 
of site usage at other times.) 

To assess whether the survey respondents were representative, we looked at the following 
behavioral measures from the transaction logs: the number of browsing sessions each person 
had during the logging period, the number of files per session, and the average session time 
length. We compared these measures for the survey responders and the survey non- 
responders. 
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1. Analysis - Version 1: t-tests 

To assess the likelihood and magnitude of response bias, we performed a series of t-tests, 
comparing the two groups on the three behavioral measures above. The t-test focuses on the 
observed means and provides an estimate of the likelihood that the difference between the 
means of the respondents and the non-respondents is due to chance (Steel and Torrie, 1980). In 
this case, a low p-value indicates that the survey responders are unlikely to be a representative 
sample of the population. We performed this analysis for both pilot test sites. 



Table 3.6: Representativeness t-tests 





Jack London 


SPIRO 


Respondent 


All users 


p-valuett 


Responders! 


All users 


p-valuett 


N 


161 


97,284 




45 


38,962 




Number of sessions (mean ± SD) 


2.2 ± 2.1 


1.5 ± 2.7 


0.003 


5 ±17 


1.4 ±4.6 


<0.0001 


Number of sessions (median) 


1 


1 


1 


1 


Number of files per session 
(mean ± SD) 


553 ± 408 


143 ± 306 


<0.0001 


242 ± 309 


68 ± 210 


<0.0001 


Number of files per session 
(median) 


460 


5 


93.5 


1 


Session length (seconds) 
(mean ± SD) 


24 ±24 


9 ±23 


<0.0001 


63 ±99 


11 ±46 


<0.0001 


21.5 


2 


Session length (seconds) 
(median) 


16 


5 



t Survey responders for whom log data are also available 

tt Likelihood that the responders' observed mean would have been obtained from a random sample (from t-test) 



For both sites, these results indicate that the users who responded to the survey were noticeably 
different from the typical site user— they used each site more frequently, and each session was 
longer and more in-depth. The p-values indicate that this difference is much greater than might 
have been expected due to randomness. The survey clearly suffers from response bias, and the 
respondents are a non-representative sample on the three measures we compared; based on 
these results, we should not generalize from the survey results to the whole population of site 
users. 

2. Analysis - Version 2: KL divergence measure 

While the t-test focuses specifically on the means of the two distributions, it is possible to 
compare the survey responders and non-responders by examining the overall distribution of 
the behavioral measures. (It is possible for two distributions to have similar means with very 
different distribution shapes.) 

Figure 3.8 shows the distribution of average session length for all Jack London site visitors and 
for survey responders. The distribution of site visitors is "right-tailed," with the majority of 
visitors having an average session length of thirty seconds or less. A small number have much 
longer sessions, up to 1800 seconds (thirty minutes). The shape of the curve for survey 
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responders is noticeably different, with disproportionately high response rates among heavy 
users — visitors with longer average sessions. Visitors with short average sessions (under one 
minute), on the other hand, were less likely to respond to the survey. 



Figure 3.8: Jack London Collection: Average session length (Histogram: all site visitors vs. 
survey responders) 




For this second measure of response bias, we used the Kullback-Leibler (KL) divergence 
measure (Kullback and Leibler, 1951) to quantify the difference between the total distribution 
and the distribution of the survey respondents on the same three empirical measures (number 
of sessions, files per session, and average session length). 

We performed a permutation test (50,000 bootstrap samples of the same size as the actual 
sample) to estimate two statistics. First, we calculated the likelihood of obtaining a "good" 
sample of the size observed (defined as a sample with a KL divergence < 0.5) at random from 
the population distribution. (A lower value indicates that, because of the shape of the 
population distribution or the low sample size, it would be difficult to obtain a representative 
sample of that size.) Second, we calculated the likelihood of obtaining a sample as good or 
better (with a lower KL divergence) than the observed sample. In this case, a low p-value 
indicates that the actual observed sample distribution is unlikely to have been obtained through 
random sampling. The results are shown in Table 3.7. 
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Table 3.7: Representativeness: KL-divergence permutation tests 





Jack London 


SPIRO 


Respondersf 

(mean±SD) 


All users 
(mean±SD) 


Probability of 
“good 
sample" 
tt 


p-value 

(probability) 

ttt 


Respondersf 

(mean±SD) 


All users 
(mean±SD) 


Probability of 
“good 
sample" 
tt 


p-valuefft 


N 


166 


97,284 






50 


38,962 






Number of 
sessions 


2.2 ± 2.1 


1.5 ± 2.7 


0.003 


0.997 


5 ±17 


1.4 ± 4.6 


0.000 


0.973 


Number of 
files per 
session 


553 ± 408 


143 ± 306 


0.000 


0.002 


242 ± 309 


68 ± 210 


0.000 


0.001 


Session 

length 

(seconds) 


24 ±24 


9 ±23 


0.011 


0.248 


63 ±99 


11 ±46 


0.329 


0.417 



t Survey responders for whom log data are also available. 

tt Probability of "good sample" = the likelihood of getting a representative sample (KL < 0.5) when selecting a random sample of 
the given size from the overall distribution. 

ttt p-value = the likelihood that a random sample of the given size from the overall distribution will be at least as representative 
(i.e., will have as low a KL divergence) as the actual sample of survey responders. 



The results of this analysis are somewhat mixed. For both sites, this analysis indicates that the 
survey respondents are unlikely to be a random sample in terms of the number of files per 
session, because respondents' sessions involved significantly more files. The differences in 
session length and number of sessions, on the other hand, are not statistically significant. In 
fact, the p-values for number of sessions suggest that the sample of respondents is a better 
match to the overall distribution than might be expected at random. 

The two representativeness analyses appear to give somewhat different results, with the t-tests 
indicating that survey respondents are non-random and the KL divergence measure suggesting 
that the sample is random. One explanation for this apparent discrepancy is that the two 
analyses are testing for different things. The KL divergence focuses on the overall shape of the 
response distribution; because the population distribution is so strongly right-tailed, a small 
sample is unlikely to be a close match. The t-test analysis, on the other hand, focuses on the 
difference between the means, rather than the whole distribution. 

Both the t-tests and the KL Divergence permutation tests suggest that the survey respondents 
are not representative of all site visitors, based on the specific behavioral metrics available for 
comparison. Since we know the sample is non-representative on these measures, it is unwise 
for us to draw any conclusions from the survey results about the characteristics of the site 
visitors overall. 

H. Discussion and Conclusions 

In this section, we have endeavored to explore the strengths and weaknesses of two widely 
used user research methods — online surveys and transaction log analysis — through a pilot test 
on two local sites. Based on these pilot tests, we attempt to recommend a set of efficient, 
effective methods that can provide useful information for the owners of educational resource 
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sites. The pilot tests helped clarify the specific techniques and metrics that provide the most 
useful insights into site usage; the tests also provided a vivid demonstration of some of the 
challenges and pitfalls in performing user research. 

These two methods encompass a variety of challenges: 

• Balancing user research goals with concerns about the privacy and confidentiality of 
user information is a challenge, particularly in educational and non-profit institutions. 

• Although website surveys are theoretically a powerful tool for understanding users' 
needs, motivations, attitudes, and satisfaction, low response rates make it difficult to 
obtain a representative sample of users. 

• Unless statistics are calculated and interpreted carefully, it is easy to misinterpret the 
results of surveys or TLA— particularly if the survey sample is non-representative. 
Misinterpretation can be counter-productive, lending credence to unwise decision- 
making. 

• Although TLA can be a useful tool for describing users' online behavior, it is of limited 
utility in describing who the users are, their goals, their offline behavior, and the 
outcomes of their site use. 

• Both web-based surveys and TLA are prone to overlook the universe of non-users. 
However, an understanding of non-users and their motivations can be extremely 
valuable for planning and development, and it can be misleading to ignore these 
concerns. 

1. Conclusions 

Understanding the usage and users of an educational website can provide valuable insights to 
facilitate better decision-making, improve site design, and support the site's target users in 
making better use of the available materials. Online surveys and TLA, when properly 
implemented, can be good, reliable methods for understanding site use. Both methods can be 
performed simply and efficiently, and the two methods offer complementary views of users and 
usage; TLA is valuable for describing the actual usage of the site, while surveys can reveal 
users' needs, motivations, attitudes, and satisfaction levels. 

After experimenting with a variety of different survey and TLA tools and analyses and 
evaluating their complexity and their usefulness, we have reached the following conclusions 
regarding the use of these tools. 

a. Surveys 

Website owners have much to gain by surveying their users. A survey of site users can be very 
revealing and provide insights into users' thoughts and feelings that are unavailable through 
other methods. Online surveys can be performed very simply, using a variety of inexpensive or 
freely available tools and websites (such as Survey Monkey or Zoomerang 102 ). 

It is important to be aware, however, that survey responses will probably not be representative 
of all site users. Think of survey results as suggestive, rather than authoritative. 



102 http://www.surveymonkey.com, http://info.zoomerang.com 
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When developing a survey, it is important to plan the analyses first, and design the questions to 
support the specific analytic goals. It is also a good idea to pilot test the survey, either online or 
in person, with a small number of known users; pilot testing is incredibly valuable for fine- 
tuning the survey questionnaire. Presser (2004) provides a helpful analysis of evaluation 
methods to use during questionnaire pilot testing. 

When designing the survey, it is important that the questions apply to first-time visitors; our 
research suggest that they are the vast majority of users. For example, avoid questions about 
what users "usually" do on the site. Also, avoid questions about results or satisfaction with the 
site, since first-time site visitors will have no experience from which to answer. Questions about 
user identity, demographics, and intentions or goals are valuable and apply equally well to new 
and return visitors. 

b. Transaction log analysis 

Transaction log analysis helps in evaluating actual site usage. A variety of software packages 
are available for TLA (see Appendix M). If the organization has the time and expertise, high- 
end packages can provide the flexibility and power to perform almost any analysis imaginable; 
however, even the free packages provide a wealth of valuable information and are much easier 
to install and run. Many of the basic analyses can be fully automated, with reports generated 
every month, for example. Other analyses may require a greater time investment from the 
researcher. 

c. Wlto are the users? 

Understanding the types and characteristics of a site's users is of interest to many site owners; 
sites designed for a particular target population may be interested to see if their target users are 
their actual users. Unfortunately, this most basic question can be difficult to answer with any 
confidence. 

A website survey can ask respondents to describe themselves; however, because the response 
pool tends to be small and non-representative, survey responses can be misleading. (It is still 
worth including some basic questions about respondents' identity, to provide a sense of who is 
responding to the survey.) 

Transaction logs can provide a few clues to users' characteristics. The number of unique IP 
addresses can provide an approximation of the number of individual users over a time period; 
this number is useful for comparison with other sites, or to track changes over time. As a first 
step toward describing these users, one can examine users' hostnames for the breakdown of 
top-level domains, paying particular attention to .edu domains and international domains. 
Although these statistics cannot tell you the actual number of education users or international 
users, they are still useful for comparative purposes: tracking changes over time or drawing 
comparisons with other sites. 

For sites affiliated with a college or university (or targeted at college and university users), it 
may also be valuable to look at the hostnames of specific schools (the second-level domain 
names). The relative proportion of traffic from different institutional domains provides only an 
approximation of the distribution of educational users; however, it may be particularly useful in 
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estimating the proportion of local use from the hosting institution, relative to use from other 
institutions. 

TLA can also be used to estimate the proportion of first-time users versus repeat users, and the 
percentage of total traffic attributable to each. Understanding the proportion of first-time users 
can guide design decisions and inform assumptions about a user's knowledge of or familiarity 
with the site. 

d. Characteristics of usage 

TLA can be useful for providing an overall sense of how a site is used. To understand the 
intensity and depth of site usage, one can calculate the average length of each session (viewing 
time or number of pages viewed). It can also be interesting to examine the distribution of times 
across the range of users (for example, see Figure 3.8, above). 

A site's total number of file hits and the breakdown of file types tend not to be useful pieces of 
information, since these numbers are heavily driven by the design of the site, rather than 
primarily by the site usage. The number of page views, rather than file hits, is a more 
meaningful statistic. 

For sites with a variety of resource types, it can be particularly useful to track the relative usage 
of different resource types or of materials from different sections of the website. Some resources 
may be used more heavily than others, or by a greater diversity of people. This information can 
guide decisions about the site's design and about adding to or developing the site's content. 

To facilitate this type of analysis, however, it is best that the site be designed with TLA in mind. 
For instance, the file structure could divide resources according to their content type, and 
navigational and decorative elements could be separated from content elements. Dynamic, 
database-driven websites should design a system for analyzing which database elements fall 
into different categories of interest. 

e. Users' goals and objectives 

The most accurate way of determining a user's goals and objectives is to ask the user, 
presumably via an online survey. Because of surveys' typically low response rates, however, it 
can be valuable to supplement the survey results with information from transaction logs. 

One way of understanding users' goals is to examine the list of referring sites, focusing on 
specific pages that link to the site and generate a lot of traffic. These pages can provide a sense 
of the mindset and intentions of users. 

Search engine referrals are particularly interesting. It is valuable to look at the full search 
queries that led people to the site (rather than a list of individual keywords), to get an overall 
sense of what users are looking for. Search queries can help answer a variety of questions about 
users' intentions: Do search queries suggest that users are looking for resources with an 
explicitly educational or instructional focus? Are users looking for specific individual 
resources, for broad packages of information, or for the site itself? Which searches lead to 
which site pages? Are there any "false positives" — pages found by users that are probably not 
what they want? What site resources are not being found by search engine queries? 
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f. Linking TLA with online surveys 

In our pilot testing, we linked survey responses to transaction log data from the same usage 
session. We found that this process required a high level of expertise and a great deal of time. 
In general, linking the two datasets is probably more trouble than it is worth for the typical 
website. For sites that have the time and expertise, this high-level analysis can provide some 
interesting insights by drawing connections between actual site usage and individual self- 
reported attitudes, objectives, and satisfaction levels. In addition, this combination can allow 
the calculation of survey response rates and estimation of survey response bias (see page 5-20, 
above). 

If one is planning to link survey responses with transaction logs, it is important to design the 
site and the survey to support the linking, with unique identifiers that will show up in both the 
usage logs and in the survey results. The data manipulations should certainly be part of the 
survey pilot test, before the full-scale survey is launched. 
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GOAL 3: WHY STUDY USERS? 

GOAL 3 A: INTERVIEWS WITH DIGITAL RESOURCE PROVIDERS 
Principal author: Shannon Lawrence 
A. Purpose and goals 

Our goal in talking to digital resource providers was to better understand the role of user 
studies and usage research in daily practice and decision-making. As such, our goals for the 
one-on-one interviews were: (1) to test our initial sort of digital resource characteristics, (2) to 
collect opinions on the importance of user research to digital resource providers, and (3) to 
determine if certain factors and their attendant digital resource characteristics (e.g., histories, 
funding models, architectures, etc.) are associated with successful strategies for integrating an 
understanding of users into development and maintenance activities. Where possible, data on 
cost and collaborative development strategies were collected. These interviews addressed the 
following specific areas: 

• Background and original purpose of a given site, including the design of a site for 
teaching, and its integration with other external resources such as course packs, 
bibliographies, etc. 

• Knowledge about site use and what methods digital resource providers use to measure 
and evaluate use 

• Funds digital resource providers have allocated to assessing use or need for assessing 
use 

• Results digital resource providers have collected about users and usage 

• Remaining questions digital resource providers have about usage, but for which they 
have not collected data 

• How digital resource providers incorporate knowledge about use with planning and 
development activities to both improve integration of their site into undergraduate 
education environments and to sustain development and maintenance activities. 

The questions we addressed to digital resource providers included: 

• Who are your intended audiences? 

• What are the educational goals of your site? 

• How do you measure use, and what is most successful? 

• What do you know about demographics, behavior, and motivations of your users? 

• Do you distinguish among various educational audiences and contexts (e.g., community 
college, Rls, AP courses, fully online vs. on-campus hybrid, international markets, 
liberal arts, vocational, etc.)? Between random users and more cohesive communities? 
Between those willing or unwilling to pay for use of online resources? 

• Where are the gaps in your current data about users, and how might those gaps be 
filled? 

• What is the relationship, if any, between your understanding of current and potential 
users, and questions of user demand and sustainability? 
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1. Terminology 

In our attempt to find a common language to describe the digital resource landscape and the 
place of user studies within that landscape, the interview process with digital resource 
providers identified a number of complexities, described in detail below. 

For the purposes of this summary, we will opt for simple terms. When describing electronic 
resources (e.g., learning objects, modules/ units, self-contained courses, images, texts, etc.) we 
will use the term "digital resource " to keep our definition as broad as possible. When we refer to 
a specific collection of digital resources, however, we will use the term "site." Some 
interviewees pointed out that the term "website" did not aptly describe the complexity of their 
collection or service, especially those that were subscription based. While we recognize the 
limitations of the term, most digital resources are (or can be) accessed through a specific URL 
and, for this summary, the term "site" provides a convenient shorthand. 

As noted above, there are at least three general categories of individuals involved with creating 
and providing digital resources: aggregators, who select and promote resources; developers, 
who shape user interactions, export mechanisms, and access paths; and content creators and 
owners, who conceive, assemble, describe, and digitize content, and may include teams across 
institutions. Because our interview participants fell into different, and often overlapping, 
categories, we use the term " digital resource provider" to describe the wide array of individuals 
interviewed. 

B. Methodology 

Between April and September 2005 we conducted thirteen telephone interviews with digital 
resource providers and two interviews with other stakeholders in this field. 103 Additionally, 
many of the interview participants, as well as other stakeholders in the digital resource arena, 
attended a meeting in May 2005 at which we were able to get critiques on our initial findings. 
Conversations that flowed from that meeting have influenced how the issues presented here are 
framed and prioritized. A detailed summary of that meeting can be found in Goal 3B. 

All interviews followed a structured protocol (see Appendix O) and were conducted by a single 
researcher to ensure consistency. In addition to questions about the measurement of usage data 
and user studies, the protocol covered various topics ranging from history and development to 
issues of financing and sustainability. The tables below are an attempt to make comparisons 
across various types of sites, though they do not wholly reflect the richness of the data. We did 
include several open-ended questions to accommodate the unique nature of individual sites at 
various stages of development. Thus some of the information reported here may not be 
generalizable to the larger digital resource landscape. 

1. Subjects 

Our subjects were not randomly selected. We built our sample of sites based on previous 
research, including the faculty survey. We initially identified sites that were robust in size, had 



103 Initial interviews were conducted with eight digital resource providers in April and May 2005. Follow-up 
interviews with five of those sites, and three additional interviews, were conducted between April and September 
2005. 
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a depth of potential user data available, and had potential classroom applications. Nine of the 
sites included in our sample are initiatives that have been funded by the Andrew W. Mellon 
Foundation and the William and Flora Hewlett Foundation, who also funded our research. The 
remaining four projects were not funded by these foundations to the best of our knowledge. 
Although the sample is small, it provides a snapshot of the diversity of free and open 
educational content available on the web. The following thirteen sites participated: 



Table 4.1: Sites interviewed 



Project 


Description 


Website 


Connexions (Rice University) 


Learning modules, units, and 
courses 


http:/ /cnx.rice.edu 


Harvard University Open 
Collections Program: Women 
Working 1800-1930 


Online archive (digitized 
historical materials) 


http:/ / ocp.hul.harvard.edu/ww 


History Matters: The U.S. Survey on 
the Web (Center for History and 
New Media, George Mason 
University and American Social 
History Project, City University of 
New York) 


Primary documents, web 
gateway, teaching materials, 
and forums 


http:/ /history matters, gmu.edu 


Humbul Humanities Hub 


Cataloged and annotated 
humanities resources 


http: / / www.humbul.ac.uk / 


Labyrinth 


Medieval studies resources 


http:/ /labyrinth.georgetown.edu 


MIT OpenCourseWare (OCW) 


Course materials 


http:/ / ocw.mit.edu 


MERLOT 


Referratory for individual 
learning objects 


http : / / www .merlot.org 


Museum Online Archive of 
California (MO AC, California Digital 
Library) 


Finding aid and infrastructure 
for digital images and 
documents 


http://www.bampfa.berkeley.edu/moac 


Open Learning Initiative (Carnegie 
Mellon University) 


Stand-alone courses in 
modular form 


http: / / www.cmu.edu/ oli 


Purdue Online Writing Laboratory 
(OWL) 


Language resources and 
interactive support 


http:/ / owl.english.purdue.edu 


RLG Cultural Materials 


Multimedia primary source 
materials 


http:/ / culturalmaterials.rlg.org 


Sharing Open and Free Intellectual 
Assets (SOFIA) 


Community college-level 
courses 


http:/ / sofia.fhda.edu/ 


Utah State University 
OpenCourseWare (USU OCW) 


Course materials 


http:/ /ocw.usu.edu 



C. Findings 

1. Testing our initial sort of digital resource characteristics 

Interviews with digital resource providers further underscored our Year 1 findings that digital 
resources are complex and difficult to compare. Our interviews began to unravel some of the 
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richness within three functional categories: 1) Site Purpose and Goals, 2) Management and 
Administration, and 3) Measuring and Understanding Site Use. 

a. Site purpose and goals 

All thirteen sites offer some degree of free and open access. In order to ensure easy access and 
remove potential barriers, all sites have a no-registration option, or no registration altogether. 
Some sites (e.g., OLI, Connexions, and MERLOT) require registration and login for members or 
subscribers to access special services (e.g., to add or manipulate materials). Registration is 
optional for History Matters and Humbul. RLG Cultural Materials, as a subscriber service, 
requires authentication for site access, but also has a spin-off resource, Trove.net, a commercial 
licensing service that provides free search capabilities for a subset of these same resources. 
Harvard Women Working, Labyrinth, MIT OCW, MOAC, OWL, and USU OCW do not have a 
registration option. 



Table 4.2: Intended audience 





K— 12 
students 


K— 12 
teachers 


College/ 

university 

students 


College/ 

university 

teachers 


Scholars 


General 

public 


Other 


Connexions 


X 


X 


X 


X 


X 


X 






X 


X 


X 


X 


X 


X 


Librarians 


History Matters 


X 


X 


X 


X 


X 






Humbul 






X 


X 


X 


X 




Labyrinth 




X 


X 


X 


X 


X 


* 


MIT OCW 


X 


X 


X 


X 


X 


X 




MERLOT 


X 


X 


X 


X 








MOAC/ O AC 




X 


X 


X 


X 


X 


** 


OLI 


X 


X 


X 


X 


X 


X 




Purdue OWL 


X 


X 


X 


X 




X 




RLG Cultural 
Materials 






X 


X 


X 


X 


X 


SOFIA 






X 


X 




X 




USU OCW 


X 


X 


X 


X 


X 


X 





X = primary, or target, audience(s); x = secondary, or additional, audiences 
* Anyone with academic training 
** Museum professionals 



All thirteen sites were developed for educational purposes, though specific goals varied. Most 
digital resource providers also had broad intentions, e.g., to provide supplementary materials 
for students, to assist instructors in teaching, or to provide general course materials to support 
any type of learning. OLI, however, also has the additional intention of providing the 
enactment of instruction. 
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All sites target postsecondary instructors as one of their primary audiences (see Table 4.2), 
although History Matters considers the K-12 audience equally important. Even those sites with 
broad outreach missions (e.g., OCW sites) recognize that their materials are often most useful 
for faculty preparing new courses. Several sites have some idea of how materials are used as a 
result of user surveys, Google link searches, and anecdotal evidence through email feedback. 

Most interviewees also claimed that their resources intend to or do reach a broad audience 
regardless of their intentions. It is possible to assume that, if resources are freely available 
online, anyone can find and access those resources using a search engine. Thus, it is likely that 
all types of users listed in Table 4.2 have accessed or will access each of the sites interviewed for 
this study, making a comparison of intended and actual users unlikely. 104 We attempted to 
collect usage data to make some comparison between intended and actual use. Unfortunately, 
because most digital resource providers interviewed do not require user registration on their 
sites, a categorical breakdown of use is not possible at this time, excepting MIT OCW and 
MERLOT, as illustrated in Table 4.3, which further illustrates the challenge that researchers face 
in understanding use. 

What digital resource providers could tell us from anecdotal evidence suggested that the actual 
audience varied significantly from the target audience in only a few cases. Connexions, for 
instance, focuses primarily on college and university instructors, but has found that, 
increasingly, some materials created on the site are being used by K-12 teachers and students. 
Interestingly, these materials receive the most hits. MERLOT reports a growing K-12 audience 
as well as education undergraduates preparing to be K-12 teachers, and other undergraduate 
students. Excepting these two cases, most sites welcomed unanticipated users but did not 
perceive the need to alter their goals or services to accommodate these new audiences, either 
because these unintended audiences were successfully using the materials as available, or 
because support services built-in as a function of the overall site (e.g., online help desks) 
represent only minor indirect costs. As the aforementioned sites both encourage or rely upon 
users to contribute content to the site, it may be reasonable to assume that the role of 
unexpected users may be more important to these types of sites. For instance, sites whose users 
produce or contribute their own content may have differing obligations to their users. 

Nonetheless, these anecdotes suggest that the divide between intended and actual audience is 
one that requires further exploration. How such unanticipated use affects a site's required 
resources, and whether or not a site should — or could — adapt to new audiences, especially 
among freely accessible sites whose funders desire support for specific target audiences, was a 
topic of discussion in some detail among digital resource providers at our May 2005 OER 
Meeting (see Goal 313). 



104 Some freely available resources may not necessarily be findable via a search engine if they are part of the "deep 
web." 
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Table 4.3: Percentage of use, by actual audience 





K-12 

students 


K-12 

teachers 


College/ 

university 

students 


College/ 

university 

teachers 


Scholars 


General 

public 


Other 


Connexions 1 
















Harvard Women 
Working 2 
















History Matters 3 
















Humbul 4 
















Labyrinth 5 
















MIT OCW 


0.8 


1.0 


28.0 


13.3 


* 


48.2 


8.7 


MERLOT 6 


n/a 


n/a 


21.0 


55.0 


n/a 


8.0 


16.0 6 


MO AC/ O AC 7 
















OLI 
















Purdue OWL 8 
















RLG Cultural 
Materials 9 
















SOFIA 
















USU OCW 10 

















* MIT does not distinguish between teachers and scholars 

1 Connexions reported 16 million hits, 1.2 million page views, and 493,000 unique visitors during October 2005. 

2 At Harvard Women Working, no consistent or predictable pattern has been established at this relatively nascent site; however, the 
total "average" per month in 2005 was 19,859 visits by 8,288 visitors. Women Working cannot distinguish between students, 
teachers and researchers. Some distinction, however, can be made between K-12 visits and college/university visits: .com, etc. 
(general public, and likely some education traffic) — 82.08% of visitor domains, 91.86% of visits; .edu (primarily 

colleges/ universities, may include some K-12 private schools) — 15.62% of visitor domains or 7.37% of visits; K-12 domains — .92% 
of visitor domains or .59% of visits; .gov (including non-U.S. .govs) — 1.38% of visitor domains or .17% of visits 

3 History Matters reported 1.2 million hits during the past twelve months 

4 Humbul serves 680,000 pages per month to around 180,000 visitors 

5 Labyrinth has no raw data as they are currently in the process of updating their servers. 

6 MERLOT does not normally report usage statistics by month. From January 1 - November 30, 2005, MERLOT reported a total of 
30,232 registered users and 758,754 visits (an average of 2,273 visits per day, with an average of 10 pages per visit). 

7 MO AC is preparing to install a new statistical package in February 2006 and will be able to collect data by domain at that time 

8 OWL's 1.8 million visits by 1.2 million computers resulted in an average of more than 4 million page views per month, over 
September and October 2005. 

9 RLG Cultural Materials reported 3661 sessions and 5148 searches per month. (Trove.net reported 20,391 unique visitors and 24,650 
visits per month.) 

10 USU has reported 7300 visits (not hits) per month since its September 2005 launch date. 

b. Management and administration 
Management and funding 

Operating budgets among sites varied, ranging from modest funding in the hundreds of 
thousands to several million dollars per year. Given the collaborative nature of most sites, often 
involving the time of individuals not directly funded by the site (e.g., faculty, institutional 
liaisons, etc.), it was difficult to extrapolate definitive numbers for budget or staff, limiting any 
further breakdown of evaluation activities as a percentage of overall activities. Therefore, the 
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information that was provided represents rough estimates. There were no obvious similarities 
between staffing models among the sites. 



Given the challenge of estimating staff, providing an approximate percentage of time/ resources 
dedicated to measuring use was even more difficult. Of the four institutions with similar 
budgets, answers ranged from "nothing" to a "part" of a staff member to 1 FTE to a rough 
estimate of 20 percent of time. Others estimated approximately 25 percent. Two institutions 
(MIT OCW and OLI) reported outsourcing use analysis to outside consultants. RLG Cultural 
Materials was unable to estimate staff time since tasks are distributed across individuals and 
departments. Several interviewees noted that more intensive use data were collected at 
particular points in the evolution of their site and thus the corresponding resources were 
equally sporadic in their distribution. 



Table 4.4: Funding sources 





Self- 

supporting 
(registration 
or licensing) 


Self- 

supporting 

(advertising) 


Institutional 

funding 




Grants - 
foundation 


Grants - 
government 


Other 


Connexions 






X 


X 


X 


X 


Trustees 


HUH 






X 




X 






History Matters 






X 




X 


X 




Humbul 






X 






X 




Labyrinth 






X 










MIT OCW 






X 


X 


X 






MERLOT 


X* 




X 






X 


** 


MO AC / OAC*** 






X 




X 






OLI 


X 




X 




X 


X 


Service 

income 


Purdue OWL 






X 










RLG Cultural 
Materials 


X 




X 




X 






SOFIA 






X 




X 






USU OCW 






X 




X 


X 





* Income from partnership/ membership fees 
** Corporate partnerships 

*** May include funding for OAC or CDL in general. Difficult to disaggregate MO AC funding from other projects within CDL. 



Nearly all digital resource providers interviewed currently operate their sites with a 
combination of institutional funding and grants, either from foundations or government 
(Table 4.4); some smaller projects depend solely on the time of volunteers. MIT OCW, USU 
OCW, and Connexions all reported some sort of private funding. RLG Cultural Materials 
reported that they receive support from membership or subscriptions, MERLOT receives fees 
from higher education and corporate partners (individual membership, however, is free to 
anyone who registers), and OLI reported modest income from for-profit efforts. Though most 
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sites in the sample were Hewlett and / or Mellon grantees, nearly all also received other funding. 
NSF and other grants received were often related to specific content development or received 
by partners or authors. Some development work for RLG Cultural Materials was funded 
through an anonymous foundation grant. Humbul was an exception in that it was primarily 
funded through JISC and AHRC as well as institutional funding, but did not receive any 
foundation funding. 105 

Expansion and sustainability 

Questions around "markets" elicited various responses. Though all sites expressed interest in 
expansion, few had formal marketing goals. Most digital resource providers mentioned that 
they paid attention to what other institutions or sites were doing. Distinct budgets for 
marketing and promotion were rare, though Harvard Women Working had placed targeted 
advertisements. Cultural Materials outsourced research for Trove.net, Humbul has a small 
budget for printing and distributing flyers and booklets, and MERLOT has a director of 
strategic development. Notably, MIT OCW has an enormous communications effort with 
several staff dedicated to outreach and awareness, including two full-time dedicated staff as 
well as the part-time efforts of two other high-level staff. Because of the scope of its mission, 
MIT OCW has a communications manager responsible for interacting with internal 
stakeholders, conferences, and media; an external outreach liaison who works with institutional 
partners that focus on MIT OCW (e.g., translation efforts, mirror sites, etc.); and two other staff 
responsible for managing relationships with other OCW efforts and communicating to high- 
level administrators. 106 USU OCW reported that they are working closely with other emerging 
OCW sites so that each site will focus first on specialty fields, then expand to general course 
offerings. The Center for History and New Media (the umbrella organization for History 
Matters) is exploring the possibility of providing their materials on mobile devices. Only 
Connexions reported being currently engaged in research for the purpose of reaching an 
identified target market. Since the time of our interviews, OLI has engaged in a market research 
project with Kevin Guthrie of the Mellon-funded Ithaka project. 

All digital resource providers relied mainly on personal contacts through academic channels as 
the primary method of reaching new markets, and many reported success. These contacts 
range from personal communication to listserv notices to the presentation and publication of 
papers about the collection. MERLOT, RLG Cultural Materials, and Connexions work with 
other sites that link to theirs. MIT has a full-time communications manager to handle public 
relations. OLI has a co-marketing agreement with McGraw-Hill for a particular course, but they 
were generally disappointed with the arrangement. MERLOT, RLG Cultural Materials, and 
OLI participate in trade shows (and Harvard Women Working did attend the American Library 
Association mid-winter meeting when the site was initially launched). MERLOT worked 
closely with course management systems, e.g.. Blackboard and WebCT, to develop build-ins for 
searching MERLOT through the course management system, and a similar effort is underway 
with Labyrinth. 



105 JISC (Joint Information Systems Committee) is the U.K.'s equivalent to the NSF. AHRC (Arts and Humanities 
Research Council) is the U.K.'s equivalent to a combination of NEA and NEH. 

106 One of MIT's explicit goals set in the original OCW plan is to help other institutions, both domestically and 
internationally, develop open-access projects of their own. In fact, most OER projects funded by the William and 
Flora Hewlett Foundation, such as the Developmental Gateway (a community site specifically for developing 
countries, see: http://www.developmentgateway.org) and David Wiley's eduCommons project (which provides 
OCW management software, see: http:/ / sourceforge.net/projects/ educommons), are working in cooperation to 
share best practices and coordinate marketing efforts. 
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About half the sites reported that their current funding models were not sustainable and that 
they were exploring alternative economic models. History Matters, Labyrinth, MERLOT, 

MO AC, and OWL indicated that while they could function on an existing base of support, 
budgetary volatility encouraged them to continuously watch for new funding opportunities. 
Nearly all digital resource providers envisioned a blended funding model involving a degree of 
institutional, foundation, and endowment support coupled with some for-profit activities. 
Particularly established sites such as Labyrinth and OWL, however, expected to rely almost 
exclusively on institutional support. RLG Cultural Materials anticipated self-sustainability 
through subscription and licensing income. 

None of the digital resource providers interviewed could point to a formal agreement with their 
funders that guaranteed permanent sustainability on any level (funding, technology, content), 
though those sites embedded within universities with already-strong university support 
believed that their sites were secure. The question of sustainability is particularly problematic 
for smaller sites that are primarily maintained by the creator, with little or no institutional 
support (especially technology support). Therefore, building communities of stakeholders 
within and outside the institution becomes important for smaller sites. 

Most interviewees agreed that the success of their site was dependent not solely upon the level 
of use, but upon a variety of factors. Most digital resource providers based at educational 
institutions felt that their site — intentionally or not — provided a value to their institution, either 
through increased public awareness or credibility, and often served unrelated needs such as 
recruitment, class enrollment, and other unexpected functions. Several interviewees described 
scenarios in which institutional administrators recognized the increased value of the site as it 
developed, resulting in increased (or secured) institutional funding. Thus digital resource 
providers at newer sites believe that institutional support would likely follow initial success and 
they plan accordingly. Membership sites, such as RLG and MERLOT, rely more on usage, 
perceived value, and scholarship support needs. 

c. Measuring and understanding site use 

Measuring use: Data collection 

We were interested in finding out what digital resource providers know about users, how they 
know what they do, and why they collect these data in the first place. 

Most digital resource providers reported that they collected various types of data on use and 
users. 107 There was concordance among all sites (except SOFIA) in the collection of web traffic 
logs, although depth of analysis varied. Additionally, most resource providers collected some 
combination of online surveys, email feedback, user search strategies, interviews, focus groups, 
anecdotal research, and/ or usability tests. Several sites conducted user studies (e.g., interviews, 
focus groups, surveys) as specific, limited-time, grant-funded projects (in some cases, these 
studies were outsourced to third-party organizations). There were no data collection methods 
on our list wholly rejected by any digital resource provider. In all cases, the level and 



107 Labyrinth, however, collected little or no data at this time due to the fact that the site is fully established (no or 
minimal new content), maintained through campus services, and otherwise unfunded. Likewise, SOFIA has not 
conducted any type of user evaluation to date. Their plan for conducting user research was part of Phase II of their 
project, which did not receive funding. 
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sophistication of both data collection and analysis depended on the choice of methods as well as 
time and budget constraints. 



Table 4.5: Ways of measuring use 





User 

registration 


Web traffic 
logs 


Online 

surveys 


Help 

questions 


Email 

feedback 


User search 
strategies 


Anecdotal 

research 


Interviews 


Focus groups 


Usability 

tests 


Other 


Connexions 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 






X 




X 


X 


X 


X 


X 


X 


X 


X 


History Matters 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 




Humbul 


X 


X 


X 


X 


X 


X 


X 






X 


X 


Labyrinth 




X 






X 














MIT OCW 




X 


X 




X 


X 


X 


X 




X 


X 


MERLOT 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X* 


MOAC 




X 


X 


X 




X 


X 






X 


X 


OLI 


X 


X 


X 


X 


X 


X 


X 


X 


P 


X 


X 


Purdue OWL 




X 






X 










p 




RLG Cultural 
Materials 




X 




X 


X 


X 


X 




X 




X 


SOFIA 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


n/a 


USU OCW 




X 


P 


P 


X 


P 


X 




X 


X 





X = data currently or previously collected 
P = data to be collected in the future 

* = MERLOT has completed a content analysis of personal collections, comments, etc. These "surrogate" measures, while not direct, 
do give a sense of how users are using the items. 



All interviewees reported collecting data about users and usage for the general purpose of 
improving the site, especially to amend content, change the design (see Tables 4.5 and 4.6), or to 
develop the site (e.g., expansion). Some sites reported, however, that this was not necessarily 
an ongoing data collection process but may have been part of usability studies during initial 
development or site restructuring. Only about half the sites interviewed depended upon these 
data to demonstrate robust usage to current or potential funders, although for at least one site 
(OLI), this is not a primary motivator. MIT OCW collects data specifically to maintain buy-in 
from internal stakeholders. They report regularly both to individual faculty with feedback from 
individual course use and from user email comments, and to higher-level administrators 
(deans, etc.) with overall usage data. This data reporting is especially important as MIT OCW 
transitions from foundation to institutional support. Few sites collected data for curiosity's sake 
alone, primarily because of the extensive resources required. Interestingly, four sites 
(Labyrinth, Harvard Women Working, OWL, and MOAC) do not collect data for development 
(e.g., assessing server size). This may be because they are institutionally supported with more 
or less static content and thus are not as concerned as other sites with expansion. 
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These varied reasons for collecting user data raise additional questions about digital resource 
providers' motivations. Conversations at our May 2005 OER Meeting (see Goal 3B) addressed 
many of this issues, such as the potential motivations for user research and especially questions 
of measuring site value and success. 



Table 4.6: Reasons for studying use and users 





Curiosity 


Defend site 
funding 


Appeal for 
funding 


Growth/ 

develop- 

ment 


Target 

marketing 

efforts 


Amend 

content 


Change 

design 


Other 


Connexions 




X 


X 


X 


X 


X 


X 






X 


X 


X 




X 




X 




History Matters 






X 


X 


X 


X 


X 




Humbul 




X 


X 


X 


X 


X 


X 




Labyrinth 








X 










MIT OCW 




X 


X 


X 


X 


X 




* 


MERLOT 


X 


X 


X 


X 


X 


X 


X 


** 


MOAC 














X 




OLI 


X 


X 




X 


X 


X 


X 




Purdue OWL 


X 


X 


X 






X 


X 




RLG Cultural 
Materials 








X 


X 


X 


X 




SOFIA 


















USU OCW 




X 




X 


X 




X 





* Internal community 

** Part of mission 

*** Learning science research 



We also asked interviewees what additional information they desired about users, and their 
responses varied. One common interest among digital resource providers was to find a way to 
measure how and for what purpose materials were being used once accessed. At least two 
sites / services wanted to know more about learning outcomes while others were interested in 
user satisfaction. RLG Cultural Materials felt that the type of classroom (or other educational) 
use was perhaps the biggest marker of value. Other desired information, however, varied by 
institution and ranged from site effectiveness (e.g., especially for non-western users) to who 
non-users were and why they did not use the resource. 

Understanding use: Data analysis and interpretation 

Sites varied on their level of knowledge about users. All interviewees except SOFIA reported 
that they did analyze (or at some point had regularly analyzed) web traffic logs in particular, 
though depth of analyses varied by site. Most sites sought information about general 
demographics (hits from types of institution, e.g., .com, .edu, etc.), but did not collect or analyze 
data at the individual level. Most described their knowledge about use as a general sense of 
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overall traffic, including hits, frequency, and origin (by country or by educational vs. other type 
of institution). Some sites were also able to track patterns of use. Several interviewees noted 
that log data are relative and provide a limited picture of use (e.g., no access to demographic 
data except from registered users). 108 

Those interviewees who had completed online surveys or interviews/ focus groups believed 
they had expanded knowledge of user satisfaction and other self-reported behaviors. Some 
interviewees were aware of how materials were being used, though the level of specificity 
varied depending on the source of the data informing this knowledge. Among those with 
anecdotal data, there was a good sense that materials were being used in the classroom, but 
they did not know the specific purpose of those materials. Other interviewees knew that 
teachers were using information in presentations or distributing information to students as 
reference materials. MIT, in particular, found that users, especially faculty, are idiosyncratic 
about how they use information. 

Most digital resource providers were generally satisfied with the current level of use of their site 
or service, but clearly wanted to expand use. At least four interviewees felt their sites were 
underutilized (Harvard Women Working, Humbul, MERLOT, and USU OCW), although 
Harvard Working Women was still a relatively new site and USU OCW had not yet officially 
launched the site at the time of the interview. 

Though most sites collect a wide array of data, several digital resource providers had difficulty 
identifying what information or method is most helpful. Sites had varied perspectives 
depending on the site content, goals, or stage of development. Some sites in early stages of 
either development or evaluation were unable to make any determination of priorities (USU 
OCW, MOAC, SOFIA). Of those with data already collected. Connexions felt that all data 
collected were useful. MIT OCW collects immense amounts of data and finds it important to 
apply external (user) data to site improvements that serve the end user. For instance, they now 
offer entire courses as zip files for easier download and are in the process of simplifying an 
underused calendar feature. 

While all sites depended heavily on web traffic log analysis, several of the interviewees pointed 
out the problematic nature of these data. History Matters and MERLOT both noted that web 
traffic logs are helpful in creating a general picture of use, but analysis is limited and can easily 
be misleading (the number of hits, for example, may or may not correlate with the repeat users 
and/ or the usefulness of the site). The biggest struggle for MERLOT has been the limitation of 
data collected by IP address. Like RLG Cultural Materials, MERLOT provides use information 
to their partners at the institutional level; when faculty access MERLOT from home, however, 
these data are not collected. RLG Cultural Materials is unique among interviewees because of 
their subscription service. While they roughly analyze usage data for subscribers, they were 
one step removed from end users. Thus, it was most useful for them to talk directly with end 
users (faculty) and with their instructional technology advisory group, because they were able 
to tease out issues and concerns not addressed by other data collection methods (e.g., logs). In 
fact, most interviewees noted that anecdotal evidence was often most helpful in making design 
and/ or content decisions. 



108 MIT OCW is a special case in which more data about user demographics appears to be possible because of 
technical infrastructure and special arrangements with server owners. 
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While most interviewees agreed that all collected data were useful or perceived to be useful in 
the long run, not all data were analyzed rigorously. In such cases, time constraints, especially 
deadlines, prevent staff from incorporating all data back into the process. Most interviewees 
reported that the scarcity of general resources (time, staff, funding) prevented them from 
extensive data analysis. MIT OCW reported that the global range of their audience results in 
logistical challenges (due to diversity in language, culture, etc.); they are inviting education 
students to conduct further research with international users so that they can be served more 
effectively. MERLOT reported that stakeholders did not demand usage data, though ongoing 
evaluation was built into their mission. 

Overall, there was no clear consensus about which types of data were most valuable, or about 
which research methodologies are best for answering particular questions. A pertinent 
question, given limited resources, is how much, and which, information about users is enough? 
This is an important area for further exploration and conversation. 

2. Digital resource providers and the importance of user research 

One area for inquiry is the degree to which specific methods are robustly matched with research 
questions. For example, our own research suggests that online surveys are frequently 
interpreted as providing high-quality user data about satisfaction and context of use, although 
we know that such surveys can be highly biased toward a small and self-selected sample, with 
response rates rarely above the low single digits (when matched against overall traffic). At both 
the May meeting and follow-up interviews, it became clear that the majority of digital resource 
providers in our small sample had no clearly defined evaluation plan that laid out the essential 
questions, the necessary metrics, and the methods by which to collect those data. 109 

In the case of log analyses, available data about user geography and institutional affiliation are 
usually quite fragmentary because of the large number of users who access via commercial ISPs 
or are otherwise of unknown origin. Such gaps in data make generalizations about the 
demographics of "users" speculative at best. 

Finally, we have found that it is not uncommon for some types of user studies (e.g., needs 
assessments, use cases, scenarios, etc.) to draw heavily on biased samples, sometime as the 
result of convenience. How the samples are chosen, sample size, and the mix of populations 
will determine the degree to which extrapolation to a larger population(s) is possible. We think 
that there is value in distinguishing between questions of user demand and usability; both are 
valid ways of thinking about users, but are quite different in terms of motivations, goals, and 
outcomes. Drawing this distinction begs the question: Is it worth thinking about whether there 
are costs to interpreting user data incorrectly or applying specific types of user data to the 
wrong question or the wrong planning stage? One common obstacle to understanding use that 
was mentioned in several interviews was the lack of consistent data, terminology, and methods 
that might enable comparison. Sharing methods, data, and evaluation technology is one 
possibility for addressing this concern (Khoo, 2005), though the complexity of coordinating such 
efforts may prove impossible. The development of a common typology and benchmarks is a 
first step in this effort that might provide substantial benefit. 



109 OLI and MIT OCW did have well-defined evaluation plans and both used third party researchers. 
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Overall, user research beyond basic needs analysis and usability testing fell into a loose 
category of peripheral and expendable activities. Motivation to engage in this type of research 
often resulted from the external funding to do so, or funders who required it. Thus, among our 
small sample, we surmise that rigorous user research is not an ongoing primary activity for 
most sites. 110 

3. The role of user research in strategic planning 

One area for further exploration is the extent to which site/ resource development should be 
predicated on any initial needs assessment or marketing study. We were not aware that this is 
an established activity among most digital resource providers, and some sites (especially 
archive/ library sites) plainly stated that it was not necessary as the transition to online access 
was simply the natural evolution of their services (see Goal 3B for further discussion of this 
topic). 

While some level of usability testing and user search strategies is evident in overall site design 
and/ or redesign, ongoing user research (and usage in general) seems to play a rather 
ambiguous role in long-term strategic planning. The extent to which user data informs the 
decision-making process was not clear, though we suspect that this factor may vary depending 
upon the scope and purpose of each individual site. 

D. Next steps 

The data from these interviews represent important areas for follow-up and further 
investigation. The questions of value and success have emerged as most pressing and were 
discussed in more depth at our Online Education Resources (OER) Meeting (see Goal 3B). 
Additionally, we think that the issue of sharing usage data across various projects has not been 
addressed. The variety of methods used, for varied purposes and goals, makes such sharing of 
results close to impossible. The need to make data more transparent is especially important for 
understanding use, given the varied measurements among sites that offer different resources in 
different ways. How more sharing can be achieved among projects is a topic that needs further 
exploration. 

Finally, similar in-depth conversations would be fruitful with digital resource providers at sites 
that extend beyond our sample. As OER transform within higher education, the need for 
ongoing conversation is apparent. 



110 Again, OLI's user-driven design strategy for ongoing research and MIT OCW's extensive data collection and 
evaluation are the exceptions to this statement. 
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GOAL 3B: SITE OWNERS AND USER RESEARCHERS MEETING 
Summary of Proceedings: May 23-24, 2005 

Principal author: Diane Harley. With Shannon Lawrence and Irene Perciali 
Introduction and Overview 

On May 23-24, 2005, the Center for Studies in Higher Education convened a meeting of 
seventeen experts for a discussion entitled "Online Educational Resources: Why Study Users?" 
The meeting was part of the UC Berkeley "Use of Digital Resources in Undergraduate 
Education" study. A complete agenda for the meeting and a list of participants can be found in 
Appendices N and O. 

The format of the meeting was relatively informal and flexible to maximize discussion. Our 
discussions covered four broad topics: 

• Codifying content and contexts 

• What do we want to know about users? How do we find out? 

• Users, user demand, and sustainability 

• What are the larger research questions and agendas that need to be addressed? 

The participants represented a variety of perspectives in the field of online educational 
resources. Their collective expertise included production and delivery of online educational 
resources, delivery of continuing education, user research, and foundation funding. The 
following organizations were represented: 

• Curricula: CMU OLI, Connexions, MERLOT, UC Irvine, MIT 

• Digital Libraries: JSTOR/ ARTstor, NSDL 

• Tools and Reuse: Carnegie Foundation, MIT, IKSME 

• Foundation and Society perspectives: ACLS, Hewlett Foundation 

The meeting, organized by Diane Harley, Vijay Kumar, and Gary Matkin, was born out of our 
collective interest in how we might clarify communication about users within the somewhat 
amorphous field of online education resources. Our purpose was to explore a number of 
questions. Most broadly, what do we know about users of online educational resources (OER)? 
What more do we need to know, and how do we find out? What, exactly, do we mean by 
online educational resources? Is it content, a course, or a certification? Is it open or restricted, 
and do users even know the difference? We asked from the start what we mean by users. The 
meeting also served as an opportunity to present for critique preliminary findings from the 
CSHE faculty survey (see page 4-20) and interviews of OER site providers (see page 6-3). 

Regarding terminology, we intentionally focused on Online Educational Resources (OER) broadly 
defined, because we know from our work that most users, and even some producers, do not 
actually make the distinction in practice between open-access and proprietary, restricted, or 
licensed sources (e.g. proprietary textbooks and databases that are licensed are used in 
combination with free resources found on the Web through Google). But because so many of 
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the participants are specifically involved with "open" online education resources (OOER), there is 
potential for confusion. We attempt to maintain the distinction throughout this summary. 

Proceedings were recorded and transcribed. The following report represents our attempt to 
summarize our rich discussions. It is not meant to be verbatim, nor presumptive of any one 
perspective or position, but rather a synthesis of the main ideas and disagreements that 
emerged, and the areas that were noted to be worthy of further exploration. It is a merging of 
many voices. 

Section One: The challenge of creating common frameworks for studying users 
A. Finding a common framework: codifying content and contexts 

The meeting began with these fundamental questions: 

• What do developers know about the demographics, behavior, and motivations of users 
and non-users of online educational resources? How do they find out? What do they 
want to know? 

• How does the variety of online educational content available (canned curricula, digital 
libraries, media sites, electronic textbooks, learning object repositories, etc.) and the 
infinite number of ways users can combine resources (e.g., individual digital objects and 
canned curricula; mixture of free and proprietary content; comprehensive and niche 
resources) influence our ability to make sense of the current and future landscape of user 
behavior and motivation? 

• Is it important for content providers to distinguish among various educational 
audiences and contexts (e.g., community college, Rls, AP courses, fully online versus 
on-campus hybrid, international markets, liberal arts, vocational, etc.)? 

• How are localization and translation of open content to be handled? Who are the 
mediators? How can we help users access and share pedagogical knowledge and 
innovations to better use OER? 

Questions about user behavior are tightly linked to questions of policy and planning. 
Descriptions provided by various initiatives about what they know or don't know about their 
users clarified that both information about users and key terms and methods for studying use 
vary greatly from one OER project to the next. Some projects have in-depth knowledge about 
some of their users but not about others (e.g. Carnegie Mellon University's Online Learning 
Initiative and MIT OCW), while other projects have little systematic knowledge about any of 
their users. The lack of a clear approach to studying users makes it difficult to coordinate 
knowledge about audiences, uses, sustainability models, or strategic planning among OER 
projects. To answer such questions and make OER useful to varied users and communities, we 
need a common framework for talking about and studying OER users. 

1. Barriers to systematic understanding of users: More attention has been paid to 
supply than to demand for OER 

A common assumption among participants representing open OER (OOER) initiatives is that a 
large quantity of OOER production and dissemination has been driven by the excitement of the 
mission— characterized by several participants as a "vision" or "campaign" with, perhaps, an 
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"evangelistic" bent. The site owners who participated in the meeting also acknowledged that 
they often have limited knowledge about who is using the content and why, and some are 
coming to realize that many OER, whether open or not, are not being used as widely as 
anticipated. The "build it and they will come" approach of OER initiatives, especially those 
born in academic institutions, has in many ways precluded systematic investigations of user 
behavior and demand. 

2. Codifying categories of content and users 

A major challenge identified by the meeting was creating a common framework to talk about 
various users and various kinds of OER. As one participant put it, "The trouble we have in 
defining our market or defining the need or defining our users is that we don't even have a 
common semantic about what we mean." Based on general agreement that we should be more 
precise about the categories of user study, the meeting posed the question of how to go about 
codifying the content and context of OER as a first step in coordinating among projects. 

Participants all underlined the importance of having clear categories to break down large terms 
like "digital resource" and "user." For instance, before deciding to study "faculty use," a 
researcher needs to decide what sort of OER to ask about and what sort of faculty to study— 
from what departments and institutions, and with what educational aims. Only with a precise 
vocabulary and a shared understanding can user studies be applied effectively in diverse 
contexts. As one participant put it, "Just to be able to qualify and inform what should be the 
kinds of questions that you'd ask of large initiatives I think is a worthwhile purpose." 
Participants hoped that such a codified framework could help make sense of the OER endeavor 
as a whole: "Some sort of structure or analytical framework through which research could be 
judged, research agendas can be formed, proposals can be evaluated, individual business plans 
could be assessed, and so forth." 

a. Differentiate among types of OER content 

The category of OER needs to be refined. Are we talking about sophisticated, carefully crafted 
curricula such as CMU OLI, syllabi and associated materials such as MIT's OCW, modular 
learning objects such as MERLOT, a repository that can be added to in chunks as with 
Connexions at Rice, or a digital library such as MOAC or RLG Cultural Materials? 

Participants began to elucidate the differences among OER products and sites. Some sites are 
significantly larger and broader in scope than others. Some sites offer referratory or portal 
services on top of content, and one participant reminded us to differentiate between "product" 
and "service," where the former is the online content, and the latter is the repository or portal. 
Then there are multiple sites that provide "online courses." While some of these course sites 
offer online instruction that stands alone, others offer online materials to supplement a course 
that takes place in a traditional classroom. Resources need to be differentiated according to "the 
extent to which the enactment of instruction is embedded in the resource itself," as one 
participant suggested. Curricular sites need to be distinguished from the multitude of digital 
library and archival collections that house massive amounts of "raw" digital resources, as well 
as from commercial media sites that offer a variety of media types from news stories to digital 
video. Aware of these and many other differences among OER sites, participants agreed that 
we should not over-generalize from a high-profile OOER site like MIT's OCW. OCW's success 



http:/ / digitalresourcestudy.berkeley.edu 



7-3 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



does offer a powerful vision of possibilities, but their precise profile is not necessarily what 
every user wants and what every OER site should duplicate. 

b. Differentiate among OER users and the contexts in which OER can be potentially used 

Students vs. faculty vs. self-learners 

Participants throughout emphasized the importance of distinguishing among student users, 
faculty users, and unaffiliated independent users, or self-learners. Because these groups use 
OER materials in very different ways, developing unique strategies for site and content design 
to meet those user needs may be essential for success. CMU OLI, for instance, designs materials 
that can, in principle, be used by everyone, but they often present the content in different 
forums that suit each unique target audience. (See page 7-17 for a more in-depth discussion 
about informal users.) 

Moreover, many, if not most, open-access OER sites are used by learners in other countries. 
Participants with global audiences explained that working with other countries involves 
adapting to a diversity of learning styles and objectives, as well as navigating the complexities 
of cultural and linguistic barriers. As one participant put it, adaptation means more than just 
translation, or even making relevant examples available. It also means that we have to 
understand the preconceptions, misconceptions, and previous learning foundations of the 
students in those countries. 

Type of institution 

Participants noted that formal academic users (students or faculty) differ based on the type of 
institution in which they work or study. Users from R1 universities, liberal arts colleges, 
community colleges, vocational schools, or high schools are likely to have differing objectives 
and access to technology. For instance, community college students (who may rely on 
computer labs more than a personal computer) often do not have the same level of access to 
technology as would students from a liberal arts college (who may have a personal laptop 
included in their tuition). Other types of users may not be in formal educational settings at all, 
but in workplaces, or learning from home. 

Users with different skill levels and learning objectives 

It is fruitful to differentiate the reasons that people use OER. Some seek a particular digital 
object for a specific purpose; some seek a completely stand-alone course; some seek 
supplemental material for a research project, assignment, or their own course; some seek to 
create their own online course or resource site; and others may just be engaged in lifelong 
learning. 

Different types of users also have different skill levels and technological sophistication. If 
technological sophistication corresponds to age, participants suggested that the fraction of 
sophisticated users will only increase. Indeed, younger students may be comfortable with 
technologies, like instant messaging, that university instructors and OER sites have barely 
begun to incorporate in their teaching methods. But the difference in skill level and technical 
sophistication is not as simple as age and can also be attributed to factors such as discipline, 
industry, and socioeconomic status. Not knowing that a resource exists is just as much of a 
barrier as not being sufficiently skilled to use it. This CSHE study, for instance, found that 
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many UC faculty simply do not know about the wide variety of restricted digital resources 
available to them through institutional licenses. 

Non-users 

One key group to study is individuals who do not currently use OER, especially if one aims to 
increase OER's audience. Why don't certain faculty, for instance, incorporate OER into their 
teaching? This research project was particularly interested in why resources weren't being 
used, and Diane Harley suggested that understanding non-users presents an opportunity. If we 
better understood the myriad reasons for non-use, including social and economic barriers, 
perhaps we could redesign OER in ways that would be useful (and that would potentially 
counteract existing disincentives and barriers). 

Category overlaps 

Finally, it is important to remember that user categories will inevitably overlap. Some users are 
also producers who create and contribute OER material of their own to existing sites. Self- 
learners encompass a vast and demographically diverse population who embark on varied 
types of independent research. Some users from one institution (for instance, a graduate school 
of education) may seek OER to be used in a different institutional context (K-12 classrooms). 
How do site producers define the user or user community, and classify users into categories, 
when communities can be so fluid and difficult to define? 

B. Codify terms and methods for studying users 

This session, which provided an overview of CSHE interviews with site owners on OER user 
studies, referenced the following questions: 

• Where are the gaps in current data about users, and how might those gaps be filled? 
What are the best methods to employ for strategic planning purposes? Should/ can OER 
projects be thinking about common questions, metrics, and approaches to 
understanding users? 

• What are consequences, if any, of poor sampling and of application of user results to the 
wrong questions? Are there consequences to not doing user demand/ market analyses 
before building content? 

• What are the best ways to leverage activities across multiple projects? 

Shannon Lawrence presented her work interviewing OER site owners (see Goal 3A). Her 
interviews, and the ensuing discussion, highlighted the problem with the current state of user 
studies: while many OER sites may use similar tools for collecting data (weblogs, online 
surveys, focus groups/ interviews), the questions asked and the precise metrics used are unclear 
and inconsistent. 

There was some concern expressed that sites may not even know about the range of possible 
evaluation tools they could use, and how to use them. At the National Science Digital Library 
(NSDL), 111 Flora McMartin found that the six digital collections under study used almost 
completely different metrics to describe themselves and their use. Similarly, Shannon Lawrence, 
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during her initial discussions with site providers for this study, found widely differing notions 
of what it means to "interview" users — it can mean conducting a focus group, a survey, an 
informal conversation, or a usability test. 

In addition to codifying the categories of OER content and OER users, participants concluded 
that it is also important to agree upon some common questions and methods for conducting 
user studies. A discussion of research design followed: some of the better-funded sites have 
hired specialists to collect and analyze their user data, but few others can afford to follow suit, 
and are left to analyze their own data without necessarily having the professional expertise to 
do so rigorously. 

Participants did not all agree that a professional-level study and analysis is always warranted. 
As Diane Harley asked, "Do we really do anything with this high-end, expensive evaluation 
work that is any better than what we would do with the informal kind of studies?" Informal 
conversations with users, it was argued, may be useful, depending on the site's objectives and 
constraints. Several of the site providers interviewed found that in-depth feedback from 
individual users was more valuable than large, comprehensive surveys. This type of informal 
user study may be especially useful during the early stages of development, when a site is 
determining direction, and later when usability issues of existing tools might be needed. It may 
be less useful if decisions are being made about investments in new and expensive initiatives. 
Participants concurred that it would be helpful to clarify which different types of studies — 
degree of formality, scale, rigor of data analysis — were best suited to particular objectives and 
particular contexts. 

1. Clarify the purpose and objectives of user studies 

Throughout the discussion of distinctions among types of content, types of users, and types of 
study, participants returned to an important consideration: a site first needs to understand its 
objectives to best select the categories it wishes to study. One participant noted: 

In order to get a handle on the impact of use and whether outcomes are being achieved, 
we need to describe the problem we're trying to solve. Because actually, how I measure 
success, and even how I continue to do product development and dissemination, has 
everything to do with "what's the problem I'm trying to solve?" 

While there are no hard and fast rules about which methods should be used, there was 
significant discussion about the need to identify the objective of the OER project and to design 
studies to improve the probability of meeting that objective. The metric a site chooses for "the 
atomic unit of the user" — institutional affiliation, desired learning outcome, skill level — 
depends on what that site wishes to learn from its findings. Conversations among different site 
producers revealed that different studies are appropriate for different OER projects. For 
example, it is important to consider whether a project is under pressure to adapt to user 
demand or has the financial resources to create content regardless of user demand or needs. 
Sites that follow a subscription model may choose to study only visitors that use the materials 
regularly and frequently, while a site such as Connexions tries to understand and serve every 
visitor. 

In her interviews with site producers. Shannon Lawrence found that many sites collect data 
without knowing what they want to learn from it. That is, site producers had difficulty 
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expressing the connection between the data collected and its usefulness, and they often lacked a 
framework for analyzing data. This fact seemed to highlight the crux of the issue: the problem 
may be not only in collecting the data but in analyzing it. In fact, several site providers openly 
admitted to collecting data with no intention of immediately analyzing it, either because they 
did not have the resources/ funding, or because they were unclear about what possible 
questions it could answer. Some debate ensued in the discussion following her presentation and 
throughout the meeting. Some shared the concern that few sites have thought through what 
data they really value and why, while others felt that it was desirable to collect as much data as 
possible even lacking a clear objective . 112 

2. Why study users? Illustrative cases and goals 

Participants stated a wide variety of goals for user studies, ranging from improving producers' 
decision-making process, to increasing the value of the broader OER endeavor. The following 
three site producers went into some detail about the particular ways in which their user study 
impacted OER project design. 

Example 1: Carnegie Mellon University Open Learning Initiative (Candace Thille) 

User studies can improve academic value, pedagogic quality, and educational outcomes. 

OLI is driven by a mission to improve educational pedagogy through cutting-edge 
computerized learning. Because OLI's goal is to improve pedagogy rather than, for instance, to 
reach as many people as possible or to encourage learning communities, their studies focus on 
learning outcomes and especially on professional development for instructors. They 
differentiate between two types of users: students / self -learners who do not otherwise have 
access to high-quality post-secondary education, and smaller institutions of higher learning that 
wish to incorporate OLI's course content and teaching methods. OLI designs their offerings 
differently for these two distinct audiences; for instance, for the latter they offer coaching and 
professional mentoring. They also study these two groups of users differently. While they 
communicate with their faculty and institutional partners through informal conversation, they 
study learners by gathering data about individual usage practices, learning styles, and 
performance, and improve their materials accordingly. They do not, however, collect data about 
user identities such as age, field of study, or institutional affiliation. They do try to understand 
the CMU faculty who contribute to the project: what keeps them motivated and how to 
encourage them to continue contributing high-quality material. In sum, OLI's efforts at user 
study are geared to maintaining a high-quality model curriculum. Some of that curriculum has 
spun into a commercial venture, Carnegie Learning, but the business side does not affect the 
ongoing research and development within OLI itself. 



112 It was suggested by one participant that a better approach in talking to site producers about user studies would be 
to ask which metrics best answer their most pressing questions. Follow-up interviews in which this new set of 
questions was posed resulted in an equal amount of confusion. Making the connection between important questions 
in user studies and the metrics that answer those questions is beyond the capacity of most evaluations at this stage in 
OER development. The exception to this experience may be those sites that have clear objectives tied to measurable 
goals. Also, those sites that invest in professional evaluation are more likely to have a comprehensive evaluation 
plan. 
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Example 2: Ithaka/JSTOR/ARTstor (Roger Schonfeld) 

User studies can improve usefulness to specific, known users and can inform choices about content and 
tools. 

ARTstor faced some choices about how to design their offerings and where to invest their 
limited resources: in improving the quality of an individual image, in offering more images, or 
in providing robust subject indexing and metadata. So they turned to a needs analysis to 
address these issues and developed a formal survey that asked "what do users want?" Because 
ARTstor aims to appeal to a broad range of faculty, the survey was administered to a wide 
variety of disciplines. And indeed, although art historians preferred a higher image quality, the 
survey found that the majority of other faculty were more interested in having a very good 
cataloging and search system. The formal market study confirmed a critical issue that ARTstor 
staff had long suspected. Having these results in hand helped to give ARTstor the confidence to 
devote further resources to its metadata, including the creation of robust "crosswalks" across 
collections allowing for the best possible user experience. 

Example 3: University of California, Irvine, Distance Learning Center (Gary Matkin) 

User studies can improve economic sustainability. 

The Distance Learning Center studies its users to create a product that will draw partners to 
commercialize it. They engage in market research and product development primarily by 
trying different prototypes. In Gary Matkin's words, "One of the things we do is we build a 
prototype product and test it on an audience, and see if they like it." Rather than engage in a 
formal needs analysis, they rely on professionals and experts who understand the field or 
industry to identify areas of need. Thus, the Distance Learning Center puts resources into 
product development first, and lets the market tell them whether the product is viable and 
attractive. Gary finds this approach more cost-effective in their case, but, as another participant 
noted, such an approach depends on the cost per unit of production. Gary acknowledged that 
other types of OER products would likely require too much initial investment, and hence would 
warrant greater initial study. 

Based on these and other conversations, three goals for user studies emerged: 

a. Goal: Improve OER content and functionality 

As all three examples show in their own way, one key goal of any user study is to enable an 
OER to be responsive to various audiences' needs. Steps to this goal can include changing 
content, adding software functionality, refining the learning experience, and/ or fixing any 
back-end problems. 

b. Goal: Assess OER value 

Value came up repeatedly in discussions about the goal of user studies: participants hoped that 
understanding use would help assess the true value of OER. To answer looming questions such 
as, "Do we keep going. . .do we keep putting money into this. . .is there value?," site producers 
need to know if people are using OER. A more precise understanding of users helps move 
considerations of value from the abstract to the tangible. Participants cited instances where an 
OER that seemed valuable because of its high-quality curriculum ends up having very little 
value for an audience, whether for lack of interest in the content as packaged, access to basic 
technology/ usability, or similarity to existing OER. User studies, thus, can provide a 
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perspective on how much a particular OER realizes its value for its users. (The value of user 
studies and their relationship to sustainability is discussed in more detail in Section 2.) 

c. Goal: Strategic planning 

Another key purpose of studying users is to inform strategic planning. Is there an audience? 
And who is the audience for whom a particular product is most valuable? By realistically 
assessing a site's value to users, user studies can help producers gauge whether their vision of 
the site is on the right track, what improvements they need to make to further their objectives, 
and whether they should adjust those objectives. User studies can also create a feedback loop 
between funders and site producers by giving funders a way to know whether a particular 
grant is meeting its objective. 

While few OER sites currently engage in the sort of strategic planning that Roger Schonfeld 
described above for ARTstor, participants imagined various scenarios where user studies could 
productively inform strategic decisions about future development and resource investment. For 
instance, if a site realizes it is primarily used by self -learners, it could design new tools that 
would be of interest to that audience. User studies could also help OER producers make 
difficult strategic choices: a site may choose to narrow its profile to serve only the key audience 
that most frequently visits it. Finally, strategic planning can help a site think about the lifecycle 
of its offerings and how to respond to trends in technology and usage patterns. Aware of the 
rapid nature of technological change, participants were concerned about how to continue to 
produce content that is useful for varying communities whose needs and skills change over 
time. 

Not all participants were equally convinced that user studies would actually help OER be more 
responsive to its users. OER suppliers may be so content-driven that, as one participant put it, 

"It doesn't matter what we say about user studies, and what information, they're still going to 
be producing these repositories just willy nilly.. . ." Another participant pointed out that 
because OER use is such a moving target, the conclusions from today's studies may be very 
poor strategic guides for the future. 

Similarly, not all participants were equally convinced that strategic planning should be the main 
objective for user study in the first place. Instead, OER sites could conceive their user studies as 
academic rather than marketing endeavors. After all, as one participant phrased it, educational 
projects such as OER have never been the sort to tailor themselves to "well-targeted, well- 
defined user demands." 

3. What methods are best, in what contexts? 

The discussion then addressed some practicalities of studying users: what are the obstacles, 
technical and conceptual, facing user studies? And what are some potential solutions and 
directions for future research? 
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Obstacles to research quality and rigor: bias, the unknowable, and lack of replicability 

Bias 

Good social science research demands time and professional expertise, and most organizations 
do not have the resources or skilled personnel to conduct rigorous research (or even more 
casual research in some cases). As a result, studies can be poorly designed and executed 
(e.g., ubiquitous lack of random sampling, lack of assessing impact of small response rates, lack 
of knowledge of existing research). In particular, participants pointed out that user studies are 
prone to bias if conducted by the site's own staff, who are very invested in the success of their 
particular site. Several participants discussed the ways in which user study risks becoming a 
"self-fulfilling prophecy" that serves to validate the producers' pre-existing sense of the site's 
value. Another participant pointed out that user studies tend to favor relationships and 
products that already exist: it is much easier to ask a site's existing users about the content they 
already use than to research non-users and explore what other content users would use if it 
existed. Techniques for measuring site usage also may have bias; for example, it would be 
misleading to measure hits right after a site has launched a new advertising campaign. Being 
aware of these potential biases, however, participants could imagine finding ways to correct for 
them, or at least acknowledge them, and still produce informative studies. 

The fuzzy unknown 

A major stumbling block to collecting good data is the difficulty of understanding the full range 
of an OER site's users. As participants shared their actual knowledge and conjectured about 
who uses their site and why, it became clear that most sites freely available on the Web get hits 
from a phenomenal range of users with very different goals and motivations. Participants cited 
the international community and K-12 teachers and students as examples. It is especially 
difficult to study users who do not register and who do not log in from recognizable 
institutions. Many open-access OER sites, in particular, do not even have a registration option, 
leaving few avenues open for understanding users at all. Sites such as OLI do not collect IP 
addresses and can rely only on anecdotal and random email messages. A site such as MERLOT 
is able to track usage only from IP addresses provided by their campus partners and is thus 
unable to learn about the larger fraction of off-campus use. (The challenges presented by 
anonymous, unaffiliated visitors are explored in greater depth below, in the section dealing 
with unintended use and users.) 

Even when it comes to known users, identifying what to measure is sometimes problematic. 
How much use qualifies a site visitor as a "user?" How do we define an "active" user? As one 
participant inquired, "Is 'active' somebody who comes on once a year and spends two or three 
hours looking for materials? Is it somebody who is there once a week?" 

Participants also noted technical and cultural barriers to studying users, especially via log 
analysis studies, which can be fraught with problems (e.g., see Goal 2B, page 5-2), and when 
studying international users. When it comes to formal and informal surveys, some participants 
suspect that what users claim in surveys is not always what they do or need in practice. This 
suspicion compounds the significant difficulty of working with exceptionally low response 
rates and biased samples when interpreting survey data culled from an on-site survey. 
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Replicability 

Finally, many participants were concerned about the fact that even a well-designed study may 
be difficult to apply to other sites or types of users due to the variation in producers, 
institutions, and users. Indeed, comparing use among projects is one of the greatest challenges 
facing user studies. Can a study about faculty at the University of Wisconsin tell us anything 
about faculty in California community colleges? A very broad study may not shed light on the 
day-to-day decisions of a particular OER site, while a very specific study may not be applicable 
to other contexts. Other participants felt, more optimistically, that any user study is a building 
block that contributes knowledge about the big picture. For example, one participant described 
several ways in which the divergence among studies still teaches us a great deal about 
segmentation among user groups. 

4. Recommendations 

a. Gather more and better user data 

Participants brainstormed ways to collect more useful information from both registered and 
non-registered users. MERLOT, for instance, will soon begin asking registered users questions 
that differentiate them according to the categories outlined above, such as institutional 
affiliation. Users could also be asked to describe what they have done with the OER material 
they found, and to provide greater detail about their needs and interests. Ideas were exchanged 
about new technologies that could track a single user throughout the site, combining his or her 
registration information, feedback, and transactions into one profile. A cookie is one solution, or 
perhaps an identifying passport that tracks a user across multiple OER sites. Participants did 
agree, however, that all such data-gathering must be voluntary, to respect users' privacy, 
especially in cases where there is fear of politically motivated surveillance. 

b. Share knowledge 

Participants underscored the importance of being more aware of one another's projects. 
Understanding the existing array of OER projects can help producers eliminate duplication, 
position a site to address unmet needs, and identify potential partners with complementary 
offerings. Knowing about other projects would also help producers expand their sense of what 
is possible and what is valuable. 

Additionally, participants would like to see usage statistics shared across projects, so that user 
data can be combined and analyzed more broadly. Some conceded, however, that projects 
which compete for funds may not be willing to share their user data freely, especially data that 
was expensive to collect and is considered proprietary. 

A different discussion underscored the importance of knowing about existing studies: while 
trying to rectify the lack of sufficient user information, we should not overlook the fact that a 
great deal of research has been done on many of the relevant topics. Some time spent 
researching existing literature and other user studies would ensure that new studies do not just 
reinvent the wheel. As one participant put it, "An hour in the library is worth twenty in the 
lab." Similarly, some participants noted that foundations already provide a sort of "marketing" 
study when they articulate their funding priorities — they have already done the work of 
speaking to relevant constituencies to assess and prioritize needs. 
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c. Create a feedback loop 

In response to concerns about whether it is possible to generalize from user studies, many 
participants pointed out that the most effective general categories and questions are those that 
emerge gradually over the course of many studies, in a continual back-and-forth between the 
micro- and macro-level. All relevant variables of study — mission, users, producers, 
sustainability— are interwoven and need to be put together piece by piece, which is best 
accomplished by an iterative analysis. 

d. Develop a community approach— methods, questions, categories 

The importance of convergence was underscored by several discussions. Developing a common 
approach to user studies would further the ability to approach OOER as a collective movement 
rather than as a series of isolated projects. A valuable goal, according to many participants, is to 
articulate general principles and standards as a community. Indeed, when asked how they 
would go about designing a research agenda for user studies of OER in general, many 
participants suggested that future studies should take place under an umbrella framework that 
ensures consistency and the capacity to generalize. A coordinated effort would then ensure that 
at the end of the study period, we would be able to draw definitive conclusions about "the 
general world of users." 

Section Two: Users, user demand, and sustainability 
A. Value and sustainability in context 

Although we devoted an afternoon specifically to the topic of sustainability, it was in fact a 
recurrent subtext woven throughout the two days of discussion. The formal discussion focused 
on the following questions: 

• What do we mean by sustainability and what are its dimensions? 

• What is the relationship, if any, between understanding current and potential users, and 
questions of user demand and sustainability of open online educational resources? 

• How important is it to distinguish between those willing and unwilling to pay for use of 
online resources (e.g., undergraduate contexts versus informal learners, etc.)? Between 
intentional users who have specific educational goals and broad-spectrum users? 

• What are the financial implications of subsidizing informal users? 

Concerns about sustainability were introduced early in the meeting during Phoenix Wang's 
discussion of the Hewlett Foundation's OER value chain. She explained that the foundation has 
moved away from the creation of content and toward considerations of access and the authentic 
use of resources in real educational contexts. 

Vijay Kumar began our focused discussion by asking us to think about value and distinguish 
among the multiple aspects of sustainability: technical, financial, and organizational. He 
pointed out that technology and infrastructure are inherently transient and variable, and that 
ultimately content needs to be designed to enable choices in complex contexts. Value, as a 
measure of sustainability, is ultimately determined by heterogeneous communities and needs to 
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be understood in each community's local context. Technological innovation should not be 
considered a value for its own sake. The questions to address include: What are local priorities? 
How do usability and complexity interact to define value in specific contexts? How do costs 
and support service needs affect value? How do various sectors (e.g., publishers, faculty, 
students, etc.) value the same resources? Finally, he suggested that the organizational structure 
within which the OER is embedded should enable the participation of different sectors. This 
need is especially true for "open" OER (OOER). As Kumar put it, "Value is determined by 
people being able to get to the value." Kumar pointed out that even expensive technologies can 
be discarded if they do not fill needs in local contexts or local cultures. 

Definitions of value and approaches to sustainability also vary according to each OER's context 
and goals, or the specific problems that an OER is trying to solve. It was argued that the only 
way to understand the value of OER— for individuals, communities, and institutions — is to 
measure its impact and its outcomes. Does OER improve educational and societal outcomes? 
And how can those outcomes be measured rigorously when so many users are "informal" and 
difficult to track? 

Following this introduction, the group spent considerable time debating the components of 
sustainability and value. It was agreed that codifying the ingredients of sustainability and the 
types of value would be necessary to answer recurring questions such as, "How do you 
measure success?" and "What makes a good OER?" A number of schemes emerged for 
breaking out these dynamic aspects of sustainability and value. We unanimously agreed on 
four key components of sustainability: 

• Curricular 

• Technical/ Infrastructural 

• Organizational 

• Financial 

B. Curricular sustainability 
1. Assuring quality — who vets? 

In the world of open-access educational resources that were represented at the meeting, quality 
and value are central concerns. Users have to trust the quality of OER content to value and 
therefore use it. Real concerns about propagating misinformation and poor quality educational 
materials are common, but there are also costs to high quality. For example, quality is 
determined to a large extent by accurate, timely, and updated information. But these very 
requirements can be a significant obstacle to sustaining OER financially. 

Who, if anyone, is responsible for assuring OER quality? Two primary answers were provided: 
the content producers should survey and control the quality of their content, or it can be left up 
to the user community to vet the material, either through peer review or some kind of ranking 
based on social tagging mechanisms. The ensuing conversation explored the contents, 
purposes, and audiences associated with each of these models. 
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Producers can vet 

There are a number of issues that relate to quality in OER. Users often rightly associate OER 
content from prestigious institutions like MIT with higher quality. Such universities already vet 
the faculty and other staff who create OER; in these cases, it is the institution that guarantees 
quality. In other cases, however, OER is not the product of an accredited higher education 
institution but rather an aggregator, referrer, or independent OER publisher. As new providers 
of OER emerge, it may not always be easy for users to sort the good from the bad. 

Some producers choose to actively control quality by strictly enforcing their own pedagogical 
and production standards, sometimes linked to highly specific course or certification 
requirements. In these cases, scope and sequence is relatively fixed. This tight linkage may 
result in such highly structured and linear content that it becomes difficult to reuse the material 
outside of the context originally envisioned by the producers. There was some discussion that 
the more independent and inexperienced the learner, the more structured the material needs to 
be, which itself creates tension between quality and reusability. 

Other sites, such as the referratory MERLOT, have established a peer review process, which 
benefits from carefully chosen content experts and a clearly defined set of criteria. In both cases, 
the OER are "branded," and that branding confers quality at a relatively high level. When 
branding is important, it also becomes paramount to avoid brand dilution. As a result, some 
contributors and content are turned away, which is seen by some as antithetical to the idea of 
"open" content. Such perceptions of elitism create additional tensions within the OOER 
movement. 

The community can vet 

At the far extreme of OER being produced by a prestigious research university such as MIT are 
sites where anyone can contribute to a corpus of information and no one will be turned away 
(e.g., the Wikipedia model). In marked contrast to the MIT OCW model, the user community 
itself takes the place of institutional or individual authority over quality. In a totally user- and 
learner-centered model, quality is a function of whether users get what they need from the 
material. Quality in this scenario is simply relative. However much work an institution puts in 
to producing high-quality OER, it is ultimately up to each user to make the final decision about 
whether the material meets his or her standards and will be useful. Connexions and Google are 
two online examples where quality is vetted by a grassroots community process: a voluntary 
peer-rating system in the former, and a mechanized system in the latter. 

Alan Wolf, meanwhile, provided a different perspective. 113 The science faculty that he studies 
claim to trust neither peer review nor community vetting; instead, they simply rely on their 
own personal judgment in every case of using an OER, or they consult with a trusted colleague. 
Similarly, the responsibility of quality could also rest with the individual faculty member who 
guides student users to the best OER for his or her purposes. In this model, though, the vast 



113 Alan Wolf described the National Science Digital Library's (NSDL) user study he and others are conducting. 
NSDL noticed that, despite the high quality of their digital collections, their collections were not being used as much 
as anticipated. In designing the study of faculty. Wolf and his colleagues followed the CSHE model. They are 
collecting existing research on this topic, and also plan to design a toolkit that would make their study replicable and 
perhaps useful to others. 
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emerging universe of unaffiliated informal users remains without the benefit of such a guide, 
and users are left on their own to make sense of the information. 

The role of (active) user communities in sustainability 

A community can be made up of users who are studying the same material or who are 
interested in the same knowledge clusters. Once connected to an OER community, users are 
more committed and likely to contribute to the site. The types of communities discussed 
ranged from students who take a course together, to self-learners who meet while studying the 
same material, to faculty in a similar discipline who might share and develop course content. 

The type of content, or the degree to which it is "cooked," may determine to some degree the 
role of the user community. The more easily the content can be manipulated, the more likely a 
user community may be given free reign over adapting materials both for their own needs and 
for the user community at large. Of course, the downside is that the independent learner is left 
with a diminished learning "roadmap" to follow. 

One model, illustrated by Connexions, encourages a user community to actively contribute to 
OER, ensuring curricular sustainability in two ways. By taking an active role in developing 
content, users help the site offer continuously growing and improving content. Simultaneously, 
an active user community can ensure that the site makes an impact: users develop the content 
that is useful to them in their local context and that takes into account their (or their students') 
learning styles and objectives, rather than what the institution or producer would unilaterally 
choose. By building user communities in this manner, OER can develop a feedback loop 
between users and site producers (or developers), but only if there is a mechanism to document 
the products and processes created by the communities in question. In the case of Connexions, 
each time a user makes changes to a set of materials, a unique version is published to the 
corpus. 

Participants recognized ongoing problems and risks when it comes to community reuse. Much 
of the time, repurposing occurs outside of the OER and does not return to its site of origin 
(unlike Connexions), and hence does not increase the overall corpus of knowledge. Some 
participants felt that reuse puts quality at risk: curricular value may get lost when pieces are 
taken out of context, and the repackaged resources that a user contributes back to the site may 
not be of high quality. Connexions has found that the pieces contributed by some users have no 
copyright clearance and/ or are not in keeping with the site producers' specifications 
(e.g., images, syllabi, course details, office hours). Additionally, including such items would 
reduce the reusability of content by others. 

Creating and maintaining community 

Though many sites find it desirable to have a critical mass of users constantly creating, using, 
reusing, and adapting content, how a site can encourage and support such vibrant growth 
remains a question. Community is dynamic and hard to predict, and it may have much more to 
do with users themselves than with anything that an institution can author. As one participant 
suggested. 

Community doesn't just happen by making content available. You really need to have 
some sort of an infrastructure, and it's not simple to do, to allow the community 
conversations to happen, and to really have leaders that get people in a subject area or 
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interest area together. They have to have some sense of belonging to that subject before 
they're ready to start giving back and improving. 

Learning communities are, as one participant put it, transient by nature: as soon as the material 
is learned or the course completed, there may be little reason to keep participating. On the 
other hand, while a particular course has an end-point, many users are ongoing learners who 
need to keep up with their field (such as tax law), and these learners benefit greatly from being 
in touch with others in that field. Communities may be easier to create around a subject area 
with multiple courses than around one course. The resulting challenge for OERs is how to 
support these various communities. 

Currently there is no common set of standard tools or practices to help achieve interactive 
community on a large scale, though emerging social computing models such as Wikipedia, 
individual "play lists," and social tagging (e.g. del.icio.us) were hailed by some as a possible 
way to merge individual needs with resource sustainability. 

Several initiatives are developing techniques for encouraging community and reuse. Lisa 
Petrides and Toru Iiyoshi described their emergent projects that are being designed to facilitate 
such communities. With Hewlett Foundation support, Petrides at ISKME 114 is creating an open- 
content portal that would act as a layer to open content resources and will point users to open 
content, particularly educational materials. The project intends to have both a top-down and 
bottom-up process, so that it will be organically driven (through tagging, etc.) and will contain 
some measure of quality vetting. The portal will point users to other people's repositories, but 
will not be a repository itself. The portal will initially be seeded with pointers to particular 
content, especially courses, from specific fields (not yet identified). Toru Iiyoshi demonstrated 
the Knowledge Media Laboratory developed at the Carnegie Foundation. The KEEP toolkit 115 is 
an open technology being deployed and tested that allows instructors to gather and use OER, 
and also to reflect upon and share best practices in the use of OER . 116 

While it is far from evident that spontaneous talking and sharing among users creates real 
educational value, a substantial knowledge community could be encouraged by an appropriate 
OER site design. One conversation explored how open-source software encourages community 
to develop organically. People in open-source software participate because they enjoy the 
prestige of being part of a community that creates create software and that improves with 
iteration. The degree to which the open software movement is analogous to open content 
creation is questionable, however. In the former, bad code simply doesn't work and cannot 
endure, whereas with academic content, misinformation can propagate uncontrollably, 
especially in non- technical, non-scientific fields . 117 



114 http://www.iskme.org 

115 http://www.carnegiefoundation.org/kml/keep/ 

116 http://www.carnegiefoundation.org/kml/keep. Similar sites and toolkits are being developed, such as COSL's 
eduCommons software (http://sourceforge.net/projects/educommons) and the Development Gateway's 
community site, which targets developing countries. 

117 At the suggestion that machine translation could offer a mechanized way to make OER more readily available to 
different cultures, it was pointed out that this could be particularly dangerous in well-crafted curricula where 
accuracy is paramount. 
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Intended vs. unintended users 

The unintended or informal user community, as opposed to those university faculty or 
matriculated students for whom the OER may have been initially produced, clearly has to be 
considered to enable understanding of the value of a resource. For many OER initiatives, the 
informal learner is an exceptionally important metric of value, even though matriculated 
students, and the faculty who teach them, are the primary targets. One question that arose time 
and again was whether the OER could or should adapt their content or services to unintended 
users. How and if these informal users should be subsidized is a problem in search of a 
solution. Significant discussion around the following questions followed: 

• How does unintended use impact a site's mission? 

• How can unintended use add value and contribute to sustainability, or is it simply to be 
viewed as a pleasant by-product of targeting a core audience of paying matriculated 
students? 

• What happens when the tables turn, and OER sites are transformed by informal users in 
ways that site producers never predicted? 

• To what extent does a site subsidize the cost of redesign to accommodate all of its users, 
including those who are unintended? 

Participants answered these questions in different ways based on their unique missions, 
primary value propositions, funding models, and institutional obligations. Some sites, such as 
MERLOT, remain committed to their original target audience and do not intend to change their 
mission or customize content for unintended users. Other sites, such as MIT OCW, view 
unintended international users as a new strategic opportunity and have aggressively adjusted 
development efforts and resources to meeting the needs of these new audiences. 

To some participants, unintended use is an opportunity for creative reuse. The pedagogical 
innovations that occur around some materials often result in uses that the original author had 
not intended. To other participants, unintended use represents a possible revenue opportunity, 
signaling to the site producer a new market that may be larger and / or more lucrative than the 
site's original target audience. 

Participants discussed reasons that an OER site should not or could not change course to serve 
an unintended audience. During a discussion exploring the pros and cons of expanding to serve 
the K-12 audience, particularly the lower grades, it became clear from Geneva Henry's 
description of Connexions that the support costs of accommodating a new audience could be 
potentially very large. Costs might include the necessary staff to develop new content and 
support educational goals and standards that are out of a producers' original domain in higher 
education. Some participants also worried about inevitable trade-offs: by shifting resources 
toward a new audience, a site could find itself under-serving its intended users. Finally, funders 
may not wish to support a target audience that differs from the audience they originally 
envisioned. And as one participant asked, given the nature of OOER, is it possible to limit how 
a site is used? How a site accommodates unintended use requires a complicated calculus that 
must account for the site's mission, scope, financial model, desired impact, quality control, and 
targeted constituencies. 
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Finally, if content can be used with other technologies and other tools, it might be reasonable to 
assume that there is another category of unintended users — commercial entities. For example, 
one question raised was how the commercial publishing community could add value to an 
open-access resource. Will the OER community resist use by such a proprietary commercial 
entity? It was suggested that we think of OER as part of a large ecosystem that includes various 
players who want to contribute value, and in doing so, help to sustain it on multiple levels. This 
notion introduces yet another tension between the values of the producer (who may want to 
give it all away) and potential users who want to integrate it into a commercial venture. 

C. Technical/infrastructural sustainability 

1. Centralized repositories, aggregation, and metasearch 

It was argued that OERs, and especially OOERs, need a common place where they can be 
reliably housed, organized, searched, and preserved, perhaps in one or more centralized OER 
repositories. And other institutions or individuals must be able to easily contribute material. 

The benefits of centralized repositories include a high level of quality across the board, 
coordinated promotion as part of national outreach, and an organization for relatively easy 
search. How a centralized repository would be organized was open to debate. Several types of 
repositories currently exist or are in development (e.g.. Connexions, NROC 118 ), but they are still 
disparate and idiosyncratic in their character, technical requirements, target audiences, and 
learning objectives. To overcome this problem, one participant suggested that materials could 
be initially concentrated into a reliable repository and then later broken up into relevant 
communities such as science or humanities. 

Aggregation of resources into "super" repositories is a supply-driven solution that may make it 
easier to maintain OER but often does not take into account what works best for users. For 
example, a centralized repository may not change the incentives and barriers of using and 
producing OER in the first place (e.g., time budgets, promotion and tenure, intellectual 
property, trust in brand). Users, perhaps especially those in academia, may want to innovate 
and use their own "collections" rather than download those that are pre-existing. Similarly, 
contributors, especially institutional contributors, may be less willing to go through the process 
of adapting and donating their content for centralized aggregation unless they feel that the 
repository is sufficiently developed and of equally high quality. 

Several participants agreed that federating searching among all OER sites would be desirable, 
though it may be technically far off, and would likely require a high level of standardization 
and collaboration among all OERs. From the users' perspective, an aggregated repository, 
where learning objects are consistently configured may more easily enable them to reuse with 
their learning management or other systems. 

2. Will Google become the master repository? 

There is plenty of evidence, including the data presented in this study, that many, if not most, 
users already treat Google as their portal to OER. There was concern expressed about the sheer 
volume of Google results, which places the impetus for sorting on the end user. As Google 
expands and digitizes more content, will it become the default gatekeeper? Yet, according to 



118 http://www.montereyinstitute.org/nroc.html 
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the this study's findings, faculty will probably not rely exclusively on Google or the books that 
Google is digitizing, but often on a vast array of free and proprietary online content, including 
their own personal collections. One participant nonetheless speculated that Google's efforts to 
allow users to store their own content may make them the one (already very large and well- 
funded) centralized OER repository. As the gatekeeper of OER, Google could not only become 
the distributor for OER resources, but likely the space where content is repurposed by new 
communities. 

3. Supporting reuse/tools 

Finally, beyond the question of where OER are housed, where would users find the tools that 
make use and reuse of the materials possible? Without such tools to easily integrate digital 
"stuff" into local contexts like a classroom, a PowerPoint presentation, a research project, or into 
different platforms, the content will likely be underutilized. Although this topic ran through 
some of our discussions as simply understood, it was not a main focus, as we were not 
concentrating on technology per se. Some technical and cultural considerations arose: 

• Faculty are particularly careful where they invest their creative energy; when they do, 
the resources they choose must be not only easily accessible but adaptable to their local 
environment. 

• One of the primary goals of OOER is to increase the corpus of overall knowledge and to 
use, not simply amplify, original materials. 

• Users may want access to a one-stop shop where they find, adapt, reuse, and integrate 
materials into their own practice, and perhaps contribute back. This process may 
require technology on all levels: a CMS on the front end, an archive on the back end, and 
a content repository at the center. 

• Well-developed products have a whole set of variables that contributed to their original 
design and therefore it may not be a good idea to make these types of products available 
for reuse. 

D. Organizational sustainability 

Sustainability is inextricably linked to organizational value, which treats the question of how 
OER fits into the organization that supports it. To what degree does the host institution value 
the OER site, and to what degree does the site's value drive institutional support? 

As discussed above, a basic concern of sustainability is an OER's physical existence: hardware, 
servers, and staff maintenance need to be continuously available. In many cases, and in contrast 
to MIT OCW, there is an ad hoc approach in which a faculty member cobbles together local 
support, be it hardware or technical support staff. If he/ she leaves the institution or runs out of 
funding, the OER can potentially be compromised. 

Long-term commitment for OER is generally unclear. Saul Fisher pointed out that "unlike print 
resources, many online or open educational resources are unmoored from a given learned 
society, or publisher. Typically, they are instead attached to an individual scholar or to a center 
that produces them, or to the host college or university for such scholars or centers." 
Institutional or organizational commitment can take many forms and can be garnered on 
several levels. 
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Depending on the OER, institutional commitment can overlap with all forms of sustainability; 
content, servers, and software may all require regular updates, and there is a cost for that. 
Further, the question of buy-in arose: at what level is institutional support sustainable? At the 
individual faculty level? By discipline or department? By school or college? Or at the highest 
level through commitment by a president or chancellor? There was general agreement that the 
success of MIT OCW is very much a product of leadership from the top and that it primarily 
serves the institutional mission, but has secondarily resulted in many positive unintended 
consequences. 

It was suggested that, at minimum, some formal agreement should be made between the 
institution and the site developer indicating the level of the institution's commitment. This 
agreement should include a definitive time commitment, whether fixed or in perpetuum. 
Similarly, a faculty member who oversees an OER should clarify that its intellectual property 
can be used by the institution even in his or her absence. Thus, organizational sustainability 
requires a two-way commitment between the site producer and the institution. 

E. Financial sustainability 

How do you fund a product that you give away? At this stage, many OERs depend on a mix of 
institutional, foundation, and corporate funding, and have few concrete plans for financial 
sustainability. Foundation, government, and institutional funding cannot be presumed to be 
reliable, so producers need to look for revenue streams from the market or other sources. 

Determining when educational content and associated technology is cost effective is not an easy 
task. Saul Fisher provided background on some early Mellon Foundation educational 
technology experiments that were unable to clearly demonstrate cost-effectiveness. His 
description follows: 

Tite Mellon CEUTT studies (Fisher and Nygren, 2000) showed that in various contexts 
the use of educational technologies does not appear to be cost-effective. While the findings 
were diverse, one general lesson from the economic side of the studies was that there is no 
reason to believe that deployment of such resources is more likely to be cost-effective than 
not, for colleges and universities. One obstacle to cost-effectiveness is the moving target. 
Instructional technologies - as well as the curricular modalities they support - have been 
in flux for over a decade, and show no signs of attaining the level of stasis we associate 
with, say, teaching in front of a classroom with chalk in hand. A further challenge is 
identifying those core qualities of such technological and curricular innovations that 
merit support as near- and mid-term innovations, independent of their immediate cost- 
effectiveness (or lack thereof). 

Tire use of open educational resources has stabilized, overall, in that every college and 
university now offers or makes use of OER to some degree. Students expect it, and 
faculty are coming to terms with it - or have done so already. And the general scheme for 
production, deployment, and use is more or less well understood. Many questions 
remain, though, not the least of which involve technological and fiscal sustainability. 

These are generally questions of how, rather than whether. Indeed, whether or not using 
open educational resources is a good idea pedagogically or cost-wise, it is clearly an 
unquestionable, even largely unquestioned, element of the higher education instructional 
landscape. This use is here and we have to accept it and deal with it in what fashion we 
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may. What remains unclear is how such resources will be sustained financially as their 
use grows larger and larger. 

1. "Business plans" in an educational context 

The conversation moved to the topic of "business" models for OER. There was no pretense 
within our small group that we had the necessary expertise to provide a thorough inventory or 
an answer in an afternoon. The general conclusion was that the models that exist all fall short in 
some way, or are very specific to particular initiatives. 

There was agreement that any business model has to first determine the production costs, 
followed by costs for delivery and sustainability. It would be necessary to identify interested 
stakeholders willing to support the project, whether users, foundations, home institutions, or 
some combination of all. All of these concerns must then be tied to the desired outcome of the 
project. 

Some sites described their current efforts to develop "business plans" to supplement or, in some 
cases replace, foundation and public funding. CMU OLI, for instance, described the dilemma 
posed in anticipation of foundation funding ending. In an effort to continuously improve and 
augment their courses, both in terms of content and technology, they have collaborated with 
Ithaka to launch a formal project investigating potential business models. 

Connexions has brought in revenue by self-publishing some of their courses as textbooks; in the 
big picture, however, neither that nor any other existing business model earns enough to keep 
them fully sustainable. They considered developing some sort of consortium or exploring other 
options. After presenting the business school faculty with their ideas, they have embarked 
upon some basic market research to better explore who their stakeholders are and to identify 
the key value in their service. 

The business model that best fits the needs of a project will, of course, depend on what is being 
sustained, whether curriculum, modules, learning objects, or other types of content. A range of 
possibilities arose, and the following models were suggested as worthy of exploration. 

MERLOT, a referratory with peer-reviewed learning objects, is based on a modified, more open 
subscription model. While its content is entirely open and free to any user, institutions can pay 
to become partners. Paying a subscription allows access to services such as faculty 
development, and subscribed institutions are also invited to become part of MERLOT's 
decision-making process. These partners are responsible for "wrapping" learning materials 
with information about how people use them, including peer review, assignments, user 
comments, and the newest feature, called Snapshots, where the authors include information 
about why the learning materials were created, how they used it, and how others might use it. 
Registered users, of which there are currently more than 22,000, also have a means to create 
personal collections, where they can create a compilation of materials that they use for a 
particular class or purpose, and then annotate these materials. Other participants noted, 
however, that at this time the benefits of partnership remain too intangible to offer significant 
value over the site's free contents. 
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The Stanford Encyclopedia of Philosophy (SEP), 119 an online open-access philosophy reference 
resource, was discussed as an interesting case. As described by a participant, the SEP has 
developed a funding model styled in part after those embraced by public radio (typically, NPR 
affiliates), where site visitors are asked to contribute voluntarily to keep the site going. This 
"public radio" component is intended to fill in the gap ($1,125 million) for their overall 
fundraising effort ($4,125 million) to establish an endowment fund. 120 The SEP's larger effort 
depends upon a partnership with the global library community to raise funds through one-time 
contributions and membership dues in exchange for continuing to offer free and open 
content. 121 

Endowment model 

There was some discussion about endowment being a successful model because raising 
endowment funds is a common practice in academia. Successfully raising such funds would 
require that the donor perceive great value in the project and that the project can demonstrate 
longevity. Whether or not such a fundraising strategy could be broadly implemented in OER at 
large remains a question due to the possibility of donor fatigue, among other things. In any 
case, soliciting donations works better when it is clear which program the donor is supporting 
and when that donor is convinced of its quality, which could be a challenge for OER sites that 
do not have an established brand or a clear and loyal user community. 

The Digital Promise (Digital Opportunity Investment Trust — DO IT), 122 is perhaps the most 
ambitious model mentioned. DO IT is a nonprofit, nongovernmental agency dedicated to 
funding innovative use of advanced information technologies in libraries, archives, museums, 
school systems, community colleges, universities, arts and cultural centers, and public 
broadcasting stations, to allow them to continue to serve their essential public purposes. The 
proposed trust will be financed by revenues earned from investing $18 billion from the 
mandated FCC auctions of the radio spectrum. 

Society subscription model 

In particular scholarly communities, such as professional societies, a move to a subscription 
model could meet with some degree of success because it builds on existing sustainable 
infrastructure. For example, those who subscribe to print journals can more easily shift to using 
that same journal online. This model becomes problematic, however, because it prevents access 
to anyone who isn't a member of that society (and pays dues), an aspect that is obviously 
antithetical to the OOER mission. 

Commercial curriculum model 

It was discussed that a centralized OER repository such as Connexions could offer raw 
curricular materials to commercial vendors who would repackage and sell them. Another 



119 http:/ /plato.stanford.edu 

120 The NEH has offered a $500,000 matching grant ($1 for every $3 raised) for library partners. See: 
http: / / plato.stanford.edu/fundraising. 

121 The Joint Information Systems Committee (JISC) is now contributing to the operating fund on behalf of the U.K. 
education community, after establishing that the SEP was a first port of call and valuable tool for the U.K. 
community. 

122 http://www.digitalpromise.org 
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possibility would be to redirect the textbook revenue stream to curricular OER. The National 
Repository of Online Courses (NROC) at the Monterey Institute for Technology and Education 
(MITE) is exploring a variation of this model. 123 MITE's business model earns revenue from low 
licensing fees similar to open-source software, but also encourages "barter" — trading access to 
existing courses in exchange for new contributions to their digital resource collection. 

OER sites could offer free content with fee-based services and tools such as course management. 
This might be a way in which OER sites could explore corporate partnerships. But for these or 
any other ideas to be profitable, OER sites need to significantly increase users. As several 
participants put it, a "critical mass" of users is essential, and currently many companies do not 
see the educational market for OER as large enough. 

Commercial education model 

The University of Phoenix 124 was discussed as an example of user-centered commercialized 
education that has reached critical mass. What can we learn from how the University of Phoenix 
does business? 

• It keeps costs low by outsourcing to teachers and tutors around the world, who are paid 
by their output. 

• It keeps customer satisfaction high by offering rapid turnaround and services to 
students. 

• It appeals to demographics that have not been well served by existing higher education, 
and students who need results-oriented rather than prestige-oriented education. 

• It uses a traditional instruction model: education is organized by degrees and courses 
with teachers and grades, with a hybrid of online and face-to-face instruction. 

• It spends 30 percent of its budget on marketing. 

• The degrees it offers are desirable and convenient, and it ensures high quality by being 
very responsive to student needs and tailoring materials to student performance. 

• Its success comes from being totally user-based: it helps students achieve their 
objectives, and the prestige of the faculty /institution doesn't matter. 

2. Commercialization vs. public good 

Is commercialization desirable, or even appropriate, for OER? There is much hesitation in 
academic circles to endorse the concepts of market research, strategic planning, and commercial 
sustainability. Academic institutions do not operate in the same way as commercial enterprises, 
and, some say, we should not necessarily expect them to. Indeed, commercialization conflicts 
with OER's central mission of free access for the public good. From this perspective, OER 
should be treated as an educational resource within a tradition of formal education, one that is 
subsidized by the institution's core mission of research, teaching, and outreach, and should not 
be translated into business terms. Students and faculty are not customers, and curricular 
materials and the faculty who use them are not products. 



123 http: / / www.montereyinstitute.org/nroc.html 

124 http://www.phoenix.edu 
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Discussing a commercial side of OER often raises general anxiety about ways in which higher 
education is being transformed into a commodity. Students' learning objectives are driven by 
convenience and by the pragmatic value of a completed course or degree, rather than by 
education for its own sake. 

The discussion concluded that the traditional teaching role of higher education institutions is to 
certify knowledge and educational quality. Perhaps by focusing on this role even in a 
commercial context, universities and OER can be financially sustainable without abandoning 
their mission. As Gary Matkin put it, 

Tltere is an issue that unrecognized, it's captured in this phrase. It's an advertising 
phrase for a continuing education organization. 'Take this course before you enroll.' 

Take this course before you enroll. Here it is, it's up, it's open. That makes a lot of sense 
to a lot of people in the world because potential students may want to find out whether 
they can get a good grade on it, or they want to find out whether you can handle it, or 
whether you have time to do it and so forth. The dirty little secret of many independent 
study operations is that there's a huge dropout rate. And people actually build financial 
models around the fact that 50 percent of the people that are enrolling in courses are 
never going to finish. 

In this type of convenience market, there are implications for how we create the course. It 
has implications for the financial model, it has implications for the user, for the user 
group, and the marketing. It also has huge implications for education and now. If there's 
an exam that certifies your knowledge, then you can take this course, but you never have 
to enroll, and you can take this exam and get what you want, then you cut out the higher 
education middleman - the teacher, the registrar, etc. Wltat I'm suggesting is that we 
have a business model in which we have all this stuff up online, and this is click here to 
enroll. Wltat we found is that people go halfway through it and they say, "I want to 
enroll in this course, sure, I can do this," click here, and they enroll. And so it's that sort 
of business model that I think. . .might be a new one to try out in the realm of OER. 

3. Limits and parameters to funding by foundations, institutions 

For those OER sites that wish to remain non-commercial entities, a combination of foundation, 
institutional, and / or corporate funding nonetheless remains the only source of financial 
sustainability. While some foundations will continue to fund innovative OER projects that have 
high curricular value, foundations increasingly ask grantees to demonstrate that their OER has 
meaningful impact among users and that it will remain sustainable after the life of the grant. 
They say upfront, "Show me the other players. I don't want to fund a single institution. I want 
to see the other players, because by virtue of hiring a collective, the likelihood of success is 
increased." 

Foundation funding today is increasingly tied to OER value. During Shannon Lawrence's 
interviews with site providers, as well as in the discussions above, well-established OER sites 
agreed that, as their sites evolved, they were better able to demonstrate value (in various forms) 
to their institution and funders, which better enabled them to receive core budgetary funding 
and grants. Many emerging OER sites agreed that they would soon have to prove their worth 
to their primary constituents to receive sustainable institutional support and funding. If an 
OER makes a significant impact on matriculated students and other key constituencies, it is 
more likely to be funded and sustained. 
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F. Measuring and establishing a user base 

In sum, understanding users, differentiating among users, responding to users, are all key to 
making OER sustainable, in any financial model. As one participant put it, "Where is the user 
base? What value do users find in this? And how do they form communities to have 
something that will sustain long-term? And until you know that, you don't know how you can 
keep it alive." Any discussion of sustainability models — how to design the OER and how to 
pay for it — rapidly turns into a discussion about users. A sustainability model has to seriously 
attend to its potential customers: will enough people use the service to make it profitable, 
sustainable, and / or meaningful in such a way that the user will pay for it or that a funder or 
institution will subsidize it? 

The question of how to do market research brings us back to a set of methodological issues: 
what parameters and categories do we use to study and talk about OER's various audiences? 
There are some methodological concerns that come up when we approach the question of user 
studies with financial sustainability in mind. 

How do you define and measure "critical mass?" If it is the number of users that makes the 
OER commercially practicable, how do you determine that number? If it is market share, of 
which market is it the share? User base can be measured against all Internet users, against the 
textbook and education market, or only against the online textbook and education market. Who 
are the potential users who need to be captured to succeed commercially? The market for OER 
needs to be defined so that it is quantifiable. For instance, if one wants to define the OER 
market as faculty, the market size can be calculated by determining the total number of 
universities and faculty affiliated. Finally, what metrics are most meaningful in commercial 
ventures? None of the readily available metrics — number of hits, unique visitors — really 
capture an OER site's potential profitability. 

How do you measure success with foundation funders? One participant suggested that 
foundations should clarify their parameters: is it more important to reach new users, to improve 
education for existing users, or just to garner a large volume of users? What if you succeed at 
impacting a population that the foundation does not care about? There was general agreement 
that each OER needs to define its own mission first, then align its goals and activities with that 
mission and focus on a targeted set of funding opportunities. Any source of funding will ask 
for some measure of value based on the OER sites' actual impact and use. 

Section Three: Imagining a research agenda 

Gary Matkin concluded the meeting by inviting participants to imagine a hypothetical scenario: 
if you had five million dollars to spend on OER research, what research projects would you 
prioritize? Participants agreed that all studies should be coordinated to use a similar set of 
terms and techniques, so that findings can be shared effectively and made generally applicable. 
Multiple research topics were suggested, but they converged on one primary and one 
secondary research priority: faculty and self-learners, respectively. 
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A. Study faculty 

After some debate, participants agreed that studying faculty would yield the highest return for 
coordinating knowledge about users across projects. Faculty use is the most common channel 
by which OER has an impact, since in most cases students use OER at the behest of their faculty 
instructor, who by definition is qualified to make decisions about curriculum. As Diane Harley 
put it, there is a long tradition of certain systems (namely, higher education) being set up to 
move students through to get some kind of certification of knowledge. "It's an economic issue, 
it's a quality of life issue, it's a social-good issue." Generally faculty have been good at being 
the mediators in knowledge acquisition and creation in higher education environments. They 
are the ones with the knowledge, they (and librarians) are the ones with the abilities to find 
relevant resources, they organize knowledge in packages and pathways, and they have 
experience in teaching subjects. By focusing on them as a domain of users, we have a high 
probability of understanding a large segment of OER use. 

Several approaches to studying faculty were proposed. We can study the ways in which 
existing faculty OER use furthers "excellence in education," as one participant put it. How do 
faculty integrate OER to improve educational outcomes? Other participants felt that studies 
should focus on faculty practices and beliefs about OER rather than on learning outcomes. Why 
do so few faculty create materials and put them online to share? Why would they not give their 
knowledge away? 

Many participants felt that the key point of study should be why so few faculty use OER. What 
are the barriers to use? Are they technical, or are they related to more difficult issues? How can 
those barriers be overcome? One participant referred to claims in Robert Zemsky's 2004 report 
"Thwarted Innovation" that startlingly few faculty have adopted OER in their teaching, which 
is of great concern to OER providers. Non-users may be harder to find and harder to study, but 
they are the most important group to understand. To study what it would take to increase 
faculty use, participants recommended designing a controlled experiment of incentives and 
rewards, or studying the motivations of faculty who have just begun using OER. 

In any case, all studies of faculty use need to take into account different faculty categories, by 
institution, location, discipline, and other considerations. 

B. Study students and self-learners 

Studying "students" was also seen as a priority, but it was agreed that it presents a different set 
of methodological problems, especially when students may not have affiliations or clear 
learning objectives. Participants were very interested in learning more about self-learners, 
because they are the group about which least is known and who are possibly the most heavily 
subsidized. At the same time, participants recognize them as the best opportunity to fulfill 
OER's mission of universal free and open-access education. Indeed, attending to this 
potentially large user group might lead OER to expand its mission beyond the traditional 
institutional structure of higher education. But it was agreed that designing a rigorous study of 
this population would be exceptionally challenging for whoever undertook it. 
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CONCLUSIONS 



Why study users? This question is at the heart of our work. There are myriad reasons cited for 
undertaking and conducting user studies. They range from improving product design and 
usability, to policing sites, to facilitating policy planning and investment decisions. 

As we think about users and potential users of online educational resources, Martin Trow's 
(1997) exhortation to "disaggregate, disaggregate, disaggregate" is particularly relevant. To 
understand their diffusion and uptake, it will be crucial to understand how "the distinctions 
between elite, mass, and universal access to higher education point to different forms of 
teaching and learning, to differences in their contexts and uses." We believe that 
acknowledging the many diverse, complex, and overlapping "ecosystems" within which open 
and other educational resources will or will not be integrated is an essential first step to 
understanding users and non-users of the many high quality digital resources available. 

For our purposes there were three interrelated questions we sought to answer. 

• How do we begin to assess whether the unique modes of scholarship and pedagogy that 
characterize the humanities and social sciences require different educational technology 
solutions than those employed in science, technical, and vocational fields? 

• How do we determine if a given resource is sustainable in a given context? 

• How can we leverage the knowledge that providers and researchers have about users so 
that it can be shared? 

Empirical comparative data on use and users are essential to addressing these and related 
questions. To that end we rigorously collected data on what targeted H/SS faculty are using or 
not using, and how the social, economic, academic, and technical factors that faculty cite as 
barriers to employing available digital tools and resources in teaching vary by discipline, 
institution type, and/ or other dimensions. We also spent considerable time assessing the 
perspectives of digital resource providers and others interested in user behavior in order to 
gauge how to make the sharing of knowledge easier. 

How do we begin to assess whether the unique modes of scholarship and pedagogy 
that characterize the humanities and social sciences require different educational 
technology solutions than those employed in science, technical, and vocational 
fields? 

By focusing our work on faculty in the humanities and social sciences, we have begun to 
develop a baseline understanding of their needs, and how disciplines vary in those needs. Such 
a baseline will facilitate future comparisons with the needs of faculty in scientific, vocational, 
and technical fields. Until comparable work is undertaken in these fields, however, direct 
comparisons will not be possible. We are delighted that a project recently funded by the NSDL, 
and run by Alan Wolf and Flora McMartin, is using much of our research design to address 
issues of use in science, technology, engineering, and mathematics (STEM) disciplines. The 
comparable research design and protocols will move us closer to understanding how solutions 
can be targeted more precisely to the varying needs of the full array of disciplines represented 
in higher education. 
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What have we learned from the faculty we talked with? They use almost every conceivable 
type of resource, many of which fall outside of what are formally called "collections" or 
"educational." Faculty from different disciplines often have different needs with regard to the 
types of resources they want, and how they ultimately use resources in educational contexts. In 
addition to these disciplinary differences, where a faculty member teaches and what their 
personal needs and experiences are influences their specific choices and challenges. 

The fact that the most cited reasons for not using digital resources was that they simply do not 
mesh with faculty members pedagogies is a very important finding, and has implications for 
those who wish to increase technology adoption by faculty. We should not expect faculty, who 
we can assume know more about teaching their subject than non-specialists, to shoehorn their 
approaches into a technical developer's ideas of what is valuable or the correct pedagogical 
approach. Tools and resources need to be developed to support what faculty do. 

Our work indicates that faculty use a variety of strategies for negotiating the digital morass. For 
most, the path of least resistance is the one usually taken— a Google search, a walk down the 
hall or an email to a colleague, a visit to the website of a trusted archive, or often one's own 
eclectic "collection" of digital stuff. What is deemed "good enough" for users will depend on 
the immediate problem; a single individual may have different standards and strategies that are 
determined by the immediate objective, time, budgets, personal and institutional equipment, 
and support staff, among other variables. Related to this issue is the large majority of faculty 
who maintain their own personal digital "collections" for teaching. Where does this material 
originate? What formats is it in? How is it stored and preserved? This wealth of material is 
simply off the radar of most institutional or commercial support providers, but it apparently 
represents a large percentage of what faculty value. 

Many faculty want to build their own reaggregated resources, using their own materials and 
mixing them with resources they have collected along the way. They are concerned about the 
significant inadequacy of the classroom technologies available to them. How to first manage 
the array of available resources, and then integrate them into teaching practice is a concern for 
those who are involved in supporting faculty pedagogies and developing useful technical tools. 
For faculty, there may be an array of tools available for collecting, developing, managing, and 
actually using resources, but the efficacy and interoperability of these tools for the immediate 
tasks that faculty need supported are questionable. 

A related challenge, for those directly providing support to faculty, is the integration of learning 
management systems (LMSs) with library resources and other course content. Current LMSs 
appear to have limited overall functionality, especially since they may not allow easy 
integration with the diversity of digital resources that faculty use. 

The challenges faced by those charged with building the future tools to reaggregate varied 
resources for easier use include: 

• The difficulty, if not current impossibility, of reaggregating objects that are bundled and 
"locked" into fixed, often proprietary resources. 

• Managing and interpreting digital rights, which may include pulling data from one 
resource for integration into another. 

• The unevenness of interface usability and aesthetics. (In some disciplines, such as art 
history, faculty may care a lot about resolution quality. Yet in other disciplines, faculty 
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may create "hodgepodge" resources, often not caring about varying resolution quality 
from one record to the next.) 

• The growing demand from users for granularity (e.g., the ability to search and find the 
one particular image or piece of text they need within an entire resource). 

• The issue of knowing about and finding digital objects. Simply put, many faculty have 
no idea about the existence of local and non-local resources, especially licensed 
resources, that may be available to them. 

As Borgman (2003) suggested, most users will at one time or another need to create personal 
digital libraries, which allow the integration of resources from diverse sources for reuse. The 
possibility of a tool with these capabilities in the near future is not clear, however, despite 
efforts to construct prototypes of such spaces. Although there are many development efforts in 
the pipeline, we have been struck by the fact that most faculty may be adrift until these 
technical promises can be fulfilled. 

Are investments in digital resource production, management, and maintenance worth 
it? 



The short answer of course is "yes," simply because we now live in a world where these 
resources are expected to be there, and many expect them to be there free for the taking. At this 
stage, however, many academic and other noncommercial providers depend on a mix of 
institutional, foundation, and corporate funding, and few have concrete plans for financial 
sustainability, nor are they completely clear on the current or prospective use of their resources 
in undergraduate educational settings. How does the vast universe and diversity of resources 
defined by users, the great variety of users, and the ubiquity of faculty personal collections, 
influence thinking about sustainability and economics of educational resources in the H/SS? 

Talking to digital resource providers, we came to understand that what they provide, and for 
whom, may represent different things to different types of users in different contexts. If we are 
to begin a productive conversation around users and use in varied educational environments, 
and about whether specific resources can be sustained in those environments, distinctions need 
to be made among types of digital resources and among types of users. A user's institutional 
affiliation, skill level, and educational goals are among the variables that need to be codified. 

Definitions of value and approaches to sustainability of resources vary according to each 
provider's context and goals. The only way to understand the value of a digital resource— for 
individuals, communities, and institutions — is to measure its impact and its outcomes. When 
we convened a group of experts interested in use and users, we all agreed that focusing on 
financial sustainability alone is overly simplistic. Sustainability needs to broken down further, 
so that curricular, technical/ infrastructural, and organizational sustainability can be factored 
into any calculus for assessing value to institutions or individuals. 

The development of user communities around open digital resources is a hot topic, with some 
suggesting that sustainability can be approached when communities contribute to and organize 
content, primarily through new social software tools and associated practices. But can this 
vision be realized while ensuring high quality (which is itself achieved only at considerable 
cost)? If providers actively control quality by strictly enforcing their own pedagogical and 
production standards, it may become more difficult to reuse the material outside the 
educational context originally envisioned. But reuse introduces its own set of tensions, most 
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notably real concerns about user communities propagating misinformation and poor-quality 
educational materials. A related issue is how or if developers can afford the costs of 
customizing their resources for audiences ranging from scholars to school children, many of 
whom are simply impossible to study because of the informal way in which they access 
resources on the web. 

No discussion about investments can ignore how the growing mass of "educational" digitized 
rich media objects meshes with established scholarly research publication models. These 
objects range from personal collections to commercial image services to traditional library 
collections and beyond. Where do personal faculty digital "collections," which cross the 
boundaries between the teaching and research realms, fit into traditional ideas about scholarly 
communication? Smith (2003) describes some of the challenges to preserving new media 
scholarship and the related economic and sustainability issues. We suspect that scholarly 
practice may be linked inextricably to pedagogical approaches in various sectors of H/SS 
teaching, and that many faculty indeed place high value on integrating their own resources and 
research into their teaching practice. If this is so, what are the implications for the learning 
object repository movement and reuse of digital resources, especially curricular materials that 
are developed for specific pedagogical goals and carry the weight of the developers' 
preconceived ideas of value? 

All of these issues become more complex when the fast pace of technological change, and the 
unpredictable introduction of new products, presentation modes, and licensing agreements, 
among other things, are factored in. Understanding users, differentiating among users, and 
responding to user communities are all key to making digital resources sustainable. 

How can we leverage the knowledge that providers and researchers have about users 
so that it can be shared effectively? 

Comparing data among digital resource projects is a significant challenge facing those who 
conduct and / or rely on user studies for decision-making. Communication among resource 
providers is the key to leveraging results and knowledge. In addition to gathering more and 
better user data, the data should be widely disseminated to help producers expand their sense 
of what is possible and what is valuable. Developing a common approach to user studies 
would allow the articulation of general principles and standards as a community. 

But before user statistics can be shared, they have to be transparent. Comparison requires some 
approximation of standardization, which is now almost nonexistent in user studies. How does 
one compare a transaction log analysis from a digital library site with a curricular materials 
site? How do we assess if visitors to a site are finding any value in the materials, and compare 
that to other sites? We suggest that we can't do these comparisons effectively until the 
categories of OER content and OER users are codified so common questions and protocols are 
agreed on. All relevant variables of study— mission, users, producers, and sustainability — are 
pertinent to such comparisons. 

The unanswered question is how or if such analyses can help those involved in resource and 
tool development build more useful systems and supportive environments. We and our 
colleagues underscore the importance of sharing usage statistics and user results openly and the 
importance of the digital resource community articulating general principles and standards 
around user studies. 
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Appendix A: Digital Resources - A Snapshot 

This list of sites is gathered from faculty survey responses, focus groups, initiatives funded by the Mellon 
Foundation and the Hewlett Foundation, and our own research. 



Africa Research Program 

http:/ /africa.gov.harvard.edu 

American Presidency Project 

http:/ /www.presidency. ucsb.edu 

American South.org 

http:/ /www. americansouth.org 

American Women 

http: / / memory.loc.gov/ ammem/ awhhtml 
AP Photo Archive 

http:/ / ap.accuweather.com/ apphoto 

Archnet (Archeology Net) 
http:/ /archnet.asu.edu 

Artcyclopedia 

http:/ /www.artcyclopedia.com 

Arts & Letters Daily 

http:/ /www. aldaily.com 

Associated Press 

http:/ /www.ap.org 

Atlantic Monthly 

http:/ /www. theatlantic.com 

Avalon Project 

http:/ /www.yale.edu/lawweb/ avalon/ 
avalon.htm 

Bartleby 

http:/ /www.bartleby.com 
Beazley 

http:/ /www.beazley. ox.ac.uk/ 

British Pathe 

http:/ /www.britishpathe.com 
Cal Poly Land 

http:/ /polyland.calpoly.edu 

California Ballot Initiatives Database 

http:/ /holmes.uchastings.edu/ cgi-bin/ 
starf inder / ?path=calinits .txt&id=webber 
&pass=webber 

California Digital Library (CDL) 
http:/ /www.cdlib.org/ 

California Heritage Collection 

http: / / sunsite.berkeley.edu/ CalHeritage 



California History Center Poundation 
http:/ /www.calhistory.org 

California Newspaper Project 
http:/ /www.cbsr.ucr.edu/cnp 

Catalonian Manuscripts 

http:/ / sunsite.berkeley.edu/catalan 

Center for American Women and Politics 
http:/ /www.cawp.rutgers.edu 

Center for History and New Media, George 
Mason University 
http:/ /chnm.gmu.edu 

Center for Land Use Information 
http:/ /clui.org 

Christian Science Monitor 
http:/ /csmonitor.com 

CIAO: Columbia International Affairs Online 
http:/ /www.ciaonet.org 

Classroom Electric: Dickinson, Whitman, and 
American Culture 
http:/ /www.classroomelectric.org 

Conversations with History 

http:/ / globetrotter.berkeley.edu/ 
conversations 

Cultura 

http: / / web.mit.edu / french/ culturaNEH 

Dartmouth Jewish Sound Archive 
http:/ / www.dartmouth.edu/~djsa 

David Rumsey Historical Map Collection 
http:/ /www. davidrumsey.com 

Die Deutsche Bibliothek Database 
http:/ /www. ddb.de 

Digital Archive: Popular American Music 
http:/ / digital.library.ucla.edu/ apam 

Dime Novel Collection 

http:/ / www-sul.stanford.edu/ depts/ dp/ 
pennies 

Early English Books Online (EEBO) 
http:/ /eebo.chadwyck.com 

Early English Prose Fiction 

http:/ / collections.chadwyck.com 
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EBSCO 

http:/ /www.ebsco.com 
ECAI Iraq 

http: / / ecai.org/ iraq 

Education Resources Information Center (ERIC) 
http:/ /eric.ed.gov 

Encyclopedia Britannica 

http:/ /www.britannica.com 

Exploratorium 

http:/ /www.exploratorium.edu 
Faulkner site 

http:/ / www.mcsr.olemiss.edu/~egjbp/ 
faulkner/ faulkner.html 

Financial Times 
http:/ /ft. com 

Foundation for the Advancement of 
Mesoamerican Studies, Inc. 
http:/ /www.famsi.org 

French National Library 
http:/ / www.bnf.fr 

GALE Eighteenth Century Collections Online 
http: / / www.gale.com/EighteenthCentury 

Galileo Project 

http:/ /galileo.rice.edu 

General Sourcebook for Medieval Economic Life 
http : / / www.f or dham. edu/ halsall/ 
sbooklj.html 

Geo-Images Project 

http:/ / geoimages.berkeley.edu 

Getty Images 

http:/ / www. gettyimages.com 

Harry Stephen Keeler Society 

http:/ / staff.xu.edu/~polt/keeler 

Healy's Literature Online 

http:/ /lion.chadwyck.co.uk 

Heritage Image Partnership 

http:/ /www. heritage-images.com 

History Matters 

http:/ /historymatters. gmu.edu 

Humbul Humanities Hub 
http : / / www . humbul .ac.uk 

Instituto Cervantes 

http:/ /www. cervantes.es 

JSTOR: The Scholarly Journal Archive 
http:/ /www.jstor.org 



Kairos Rhetoric 

http:/ /english.ttu.edu/kairos 

Kite Aerial Photography 

http:/ / arch.ced.berkeley.edu/kap 

Labyrinth 

http: / / www.georgetown.edu/labyrinth 

Latin American Network Information Center 
http:/ /lanic.utexas.edu 

Leuven Database of Ancient Books 
http:/ /ldab.arts.kuleuven.ac.be 

Lexis Nexis 

http:/ /www. lexisnexis.com 

Library of Congress American Memory Project 
http:/ /memory. loc.gov 

Library of Congress Century of Lawmaking 
http:/ / memory.loc.gov/ ammem/ amlaw 

LUCI: Library of the University of California 
Images 

http:/ /vrc.ucr.edu/luci 

Mark Harden's Artchive 
http:/ /www.artchive.com 

MERLOT (Multimedia Educational Resource for 
Learning and Online Teaching) 
http:/ /www. merlot.org 

Miguel de Cervantes Digital Library / Biblioteca 
Virtual Miguel de Cervantes 
http:/ /www. cervantesvirtual.com 

MIT Free Open Souce Research Community 
http:/ /opensource.mit.edu 

MIT OpenCourseWare 
http:/ /ocw.mit.edu 

MITHologies 

http: / / www.mith.umd.edu/ mithologies 

National Election Studies 

http:/ /www.umich.edu/~nes 

New Deal Library 

http:/ /newdeal.feri.org 

New York Times 

http:/ /www. nytimes.com 

OAC: Online Archive of California 
http:/ /www.oac.cdlib.org 

Online Reference Book for Medieval Studies 
http:/ /www. the-orb.net 

Open Collections Program 
http:/ / ocp.hul.harvard.edu 
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Oyez 

http:/ /www.oyez.org 

Patrologia Latina 

http:/ /pld.chadwyck.com 

Perseus Digital Library 

http:/ /www.perseus.tufts.edu 

Personal website of authors and literary 
movements 

http: / / www.wsu.edu/~campbelld/ amlit/ 
aufram.html 

Places Online 

http:/ /www.placesonline.org 

Project Gutenberg 

http:/ / www. gutenberg.org 

Project Muse 

http: / / muse.jhu.edu 

ProQuest 

http:/ /proquest.com 
Quia 

http:/ /www.quia.com/ 

Rhizome 

http:/ /rhizome.org/ 

RiverWeb 

http:/ /www. riverweb.uiuc.edu 

RLG Cultural Materials 

http:/ / culturalmaterials.rlg.org 

San Jose State's Image Repository 
http:/ /worldart.sjsu.edu 

Saskia Ltd 

http:/ /www. saskia.com 
Science 

http:/ /www. science.com 
ScienceDirect 

http:/ /www. sciencedirect.com 

Smithsonian 

http:/ /www.si.edu 

Sor Juana Inez de la Cruz Project 

http: / / www. dartmouth.edu / ~sorjuana 

SPIDER 

http: / / eee.uci.edu/ programs/ spider 
SPIRO 

http:/ /www.mip.berkeley.edu/ spiro 

Stanford Encyclopedia of Philosophy 
http:/ / plato.stanford.edu 



Statewide Database 

http:/ /swdb .berkeley.edu 

Strindberg Museum 

http: / / www.strindbergsmuseet.se/ 
index_eng.html 

Strunk and White Pirst Original Edition 
http:/ /www.bartleby. com/141 

The Art Museum Image Consortium (AMICO) 
http:/ /www. amico.org 

The British Museum 

http: / / www.thebritishmuseum.ac.uk 

The Connexions Project 
http: / / cnx.rice.edu 

The English Poetry Full-Text Database 
http:/ / etext.lib.virginia.edu/ epd 

The Getty Collections 

http: / / www.getty.edu/ art/ collections 

The Jack London Collection 
http:/ /london.sonoma.edu/ 

The Library of Congress 
http:/ /www.loc.gov 

The Martin Luther King Jr., Paper Project 
http:/ /www.stanford.edu/ group/King 

The Museum of Unworkable Devices 
http:/ /www.lhup.edu/~dsimanek/ 
museum/ unwork.htm 

The Nation 

http:/ /www. thenation.com 

The Online Medieval and Classical Library 
http:/ / omacl.org 

The Open Video Project 

http:/ /www.open-video.org 

The Quiz Center 

http:/ / school.discovery.com/ quizcenter 

The Society for Political Methodology 
http:/ / polmeth.wustl.edu 

The Teaching Resources Center Online Writing 
Project 

http:/ / cai.ucdavis.edu/ trc/ trcdefault.html 

The Valley of Shadow 

http:/ /valley.vcdh. virginia.edu 

Thesaurus Linguae Graecae 
http:/ /www. tlg.uci.edu 

THOMAS 

http:/ /thomas.loc.gov/ 
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UBU Web 

http:/ /www. ubu.com 
UCSB library 

http:/ /www. library, ucsb.edu 
USGS 

http:/ /www.usgs.gov 

Virtual Data Center 
http:/ / thedata.org 

Visual Sourcebook for Chinese Civilization 
http:/ / depts.washington.edu/ chinaciv 

Voice of the Shuttle 
http:/ /vos. ucsb.edu 

Wall Street Journal 

http:/ /online, wsj.com 

Warner Brothers 

http:/ /www. warnerbros.com 

Web Gallery of Art 
http:/ /www. wga.hu 

Will Connell Collection Online 

http:/ / photo.ucr.edu/ photographers/ 
connell 

Women and Social Movements 

http:/ /womhist.binghamton.edu 

World Lecture Hall 

http:/ / www.utexas.edu/world/lecture 
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Appendix B: Site Owner Advisory Group 

Meeting held at the Center for Studies in Higher Education, UC Berkeley, May 17, 2004 

Arnold Arcolio, Research Libraries Group, Red / Green Light Project (Information Architect) 

Laine Farley, California Digital Library, Digital Library Services (Director) 

Diane Harley, UCB, Center for Studies in Higher Education, Higher Education in the Digital 
Age (Director), Digital Resource Study (Principle Investigator) 

Alison Head, UCB, Center for Studies in Higher Education, Digital Resource Study (Research 
Analyst) 

Jonathan Henke, UCB, Center for Studies in Higher Education, Digital Resource Study 
(Research Analyst) 

Elizabeth Losh, UCI, Humanities Core Course (Writing Course Director), SPIDER - Shared 
Pedagogical Initiative: A Database of Electronic Resources for the University of California 
Community 

Flora McMartin, MERLOT, Membership Services and Evaluation (Director) 

Rick Rinehart, UCB, Art Practice, Art, Technology and Culture (Online Curator), Berkeley Art 
Museum & Pacific Film Archive, Digital Media (Director) 

Clarice Stasz, Sonoma State University (History Professor), the Jack London Collection (Content 
Developer) 

Roy Tennant, California Digital Library (User Service Architect), the Jack London Collection 
(Site Developer) 

Glen Worthey, Stanford University, Humanities Digital Information Services (Project Director) 
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Appendix C: CSHE/CDL Educational Technology & Librarian Meeting 
Participants 

Meetings held at UC Berkeley, March 22, 2004, and May 10, 2004 

Peter Brantley, California Digital Library, Digital Library Technologies (Director) 

Barbara Davis, UCB, Undergraduate Education (Assistant Vice-Provost) 

Elizabeth Dupuis, UCB Library, Instructional Services (Head) 

Victor E. Edmonds, UCB, Educational Technology Services (Director) 

Janice L. Eklund, UCB, Visual Resources History of Art Department (Principal Museum 
Scientist, Curator) 

Mara Hancock, UCB, Educational Technology Services, Learning Systems (Associate Director) 

Gary Handman, UCB, Moffitt Library, Media Resource Center (Head) 

Michael T. Hardie, UCB, Educational Technology Services (Grants Coordinator) 

Diane Harley, UCB, Center for Studies in Higher Education, Higher Education in the Digital 
Age (Director), Digital Resource Study (Principle Investigator) 

Alison Head, UCB, Center for Studies in Higher Education, Digital Resource Study (Research 
Analyst) 

Jonathan Henke, UCB, Center for Studies in Higher Education, Digital Resource Study 
(Research Analyst) 

Oliver Heyer, UCB, Educational Technology Services (Programmer, Analyst II) 

Patricia Iannuzzi, UCB Library (Associate University Librarian) 

Rosalie Lack, California Digital Library (Evaluation and Instruction Analyst) 

Pat Maughan, UCB Library (Librarian) 

Flora McMartin, MERLOT (Membership Services and Evaluation Director) 

Ian Miller, UCB, Center for Studies in Higher Education, Digital Resource Study 
(Programmer/ Analyst) 

Merrilee Proffitt, Research Libraries Group (Program Officer) 

Jeff Rusch, UCB, Educational Technology Services (Programmer, Analyst III) 

Maryly A. Snow, UCB, Architecture Slide Library (Librarian) 

Roy Tennant, California Digital Library, Jack London Collection (User Services Architect) 
Gunter Waibel, Research Libraries Group (Program Officer) 
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Appendix D: Faculty Survey - Instrument 



Digital Resources Survey 



Thank you for participating in this important study. Hus survey asks a variety of simple questions about 
your use of digital resources in teaching undergraduates. There are no right or wrong answers; we are 
simply interested in your opinion and your experiences. 

To thank you for completing the survey, we will enter you in a drawmg to win one of seven prizes, 
including a new Palm Pilot, a £300 gift certificate from Amazon.com, or one of five £100 gift certificates 
from Amazon.com. Your odds of winning depend on die number of responses received and are 
approximately one in 75. 

Please be assured of complete confidentiality. (We ask that you do not mention your name on the survey 
so your identity remains confidential. ) The survey should take about 15-20 minutes to complete. 
Completion of this survey is voluntary, and you can skip any questions or withdraw at any time. There 
are no direct risks or benefits for you in doing this study; however, there is the possibility of a great 
benefit to undergraduate teaching. 

This research project is funded by the Andrew W. Mellon Foundation and the William & Flora Hewlett 
Foundation, and is based at the Center for Studies in Higher Education at the University of California, 
Berkeley. This research is overseen by the appropriate institutional review boards for the protection of 
human subjects. For more information about this project, please see 

http:/ / digitahesourcestudy.berkeley.edu or contact Diane Harley, Ph D , at the Center for Studies in 
Higher Education, (510) 642-5040, hedaSuclink.berkeley.edu. 



Teaching Background 



1. In what academic discipline(s) do von teach under graduates? 

(Check all that apply.) 

_1 None: I never teach undergraduate ^ Anthropology 



Languages & Literature: 

^ English 

^ Languages 

Please specify: 

^ Literature 

Please specify: 

^ Writing or composition 
^ Other: 



^ Archaeology 
O Architecture 
_] Art history 

^ Art practice, visual art, other art 
Please specify: 

Q Classics 

^ Geography 
Q Historv 
Q Political science 
Q Other 

Please specify: 
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2. What specific courses do you teach on a regular basis? 




3. What type of course do you typically teach? 

( Check all that apply.) 

d Seminars / small discussions 
LJ Labs / studio courses 

□ Small lectures (fewer than 25 students) 

Q Medium lectures (25-50 students) 

Q Large lectures (50-100 students) 

□ Very large lectures (> 100 students) 

4. Some courses are offered completely or partially online. In what course formats do you 
typically teach? 

(Check all that apply.) 

Q Online coruses (where students do not have to come to class, but do all of their 
assignments online) 

□ Hybrid courses (where students still come to class, but do many of their assignments 
online) 

□ Traditional courses (where none of the work is completed online) 

What do we mean by digital resources? 

Our definition of digital resources is intentionally broad. Digital resources... 

• may include audio, photos, maps, text, manuscripts, graphs, slides, charts, video, curricular support 
materials, or primary source materials. 

• may be either your own or others' online resources. 

• may be from library and museum collections. 

• may be from your own personal collection. 

• may be materials you, colleagues, or others have made available in an online format. 



http:/ / digitalresourcestudy.berkeley.edu 



10-9 








Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Digital Resources Survey 










page 3 of 14 




What Digital Resources Do You Use? 


5. Please indicate how often you use or have used the following types of digital 
in your undergraduate teaching: 


resources 


Types of resources: 


Almost all 
the time 


Often 


Sometimes 


Rarely 


Neva- 


Images or visual materials (drawings, 
photographs, art, posters, etc.) 


□ 


□ 


□ 


□ 


. . 


Maps 


□ 


□ 


□ 


□ 


. • 


Simulations or animations 


□ 


□ 


□ 


□ 


□ ' 


Digital film or video 


□ 


□ 


□ 


□ 


□ ' 


Audio materials (speeches, interviews, 
music, oral histories, etc.) 


□ 


□ 


□ 


□ 




Digital facsimiles of ancient or historical 
manuscripts 


□ 


□ 


□ 


□ 


. . 


Online or digitized documents 
(including translations) 


□ 


□ 


□ 


□ 




Government documents 


□ 


□ 


□ 


□ 


. . 

. . . 


Data archives (numeric databases; e g., 
census data) 


□ 


□ 


□ 


□ 




News or other media sources and 
archives 


□ 


□ 


□ 


□ 


□ 


Online reference resources (e.g., 
dictionaries) 


□ 


□ 


□ 


□ 




"Portals" that provide links or URL's 
relevant to particular disciplinary 
topics 


□ 


□ 


□ 


□ 




Personal online diaries (e.g., blogs) 


□ 


□ 


□ 


□ 


□ 


Online class discussions (including 
archived discussions) 


□ 


□ 


□ 


□ 




Curricular materials and websites that 
are created by other faculty and/or 
other institutions (e g., MIT 
OpenCourseWare, World Lecture 
Hall, Merlot) 


□ 


□ 


□ 


□ 


" ' □ ' 


Digital readers or coursepacks 


□ 


□ 


□ 


□ 


□ 


Other types of resources. 
Please specify: 


□ 


□ 


□ 


□ 


. .^. . 
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Where You Find Digital Resources 


6. How often do you use digital resources in your undergraduate teaching from each of the 
following sources? 


Sources of resources 


Almost all 
ihe time 


Often 


Sometimes 


Rarely 


Never 


Search engines / directories (e g., 
Google, Yahoo) 


□ 


□ 


□ 


□ 




My own personal collection of digital 
materials 


□ 


□ 


□ 


□ 


□ 


Public (free) online image databases 


□ 


□ 


□ 


□ 


. .^. . 


Commercial image databases (e g., 
Saskia, AMICO) 


□ 


□ 


□ 


□ 




Campus image databases from my own 
institution (e g., departmental 
digital slide library ) 


□ 


□ 


□ 


□ 


□ 


"Portals" that provide links or URL's 
relevant to particular disciplinary 
topics 


□ 


□ 


□ 


□ 


□ 


Online exhibits (e g., from museums) 


□ 


□ 


□ 


□ 


. .^. . 


Library collections (digital) 


□ 


□ 


□ 


□ 


. . 


Online journals (e g., JSTOR) 


□ 


□ 


□ 


□ 




Media sites (e g., NPR, New Y'ork 
Times, CNN, PBS) 


□ 


□ 


□ 


□ 




Other sources of digital resources. 

Please specify: 


□ 


□ 


□ 


□ 


□ 


7 . How t often have you heard about sources 


of digital 


resoiuces from each of the following? 




Almost all 
the time 


Often 


Sometimes 


Rarely 


Never 


Professional societies or discussion lists 
(e.g., H-Net, Humanist Discussion 
list, etc.) 


□ 


□ 


□ 


□ 


□ 


Recommendation from a campus 
librarian 


□ 


□ 


□ 


□ 


□ 


Word of mouth from colleagues 


□ 


□ 


□ 


□ 


□ 


Word of mouth from students 


□ 


□ 


□ 


□ 


. .^. . 


A campus department devoted to 
instructional technology (e.g., 
media or teaching and learning 
center) 


□ 


□ 


□ 


□ 




Other. . 
Please specify: 


□ 


□ 


□ 


□ 


□ 
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What Specific Digital Resources Do You Use? 



8. Please name the specific digital resources or collections that you find most valuable to 
your teaching: 




9. How often do you do each of the following activities with digital resources? 





Ai most all 
the time 


Often 


Sometimes 


Rarely 


Never 




I gather or maintain my own 
collection of digital 
resources. 


□ 


□ 


□ 


□ 


□ 




If yes... 


Almost all 
the time 


Often 


Sometimes 


Rarely 


Never 


Does not 
I apply tone 


I make my own digital 
resources available to 
others via the World- 
Wide Web. 


□ 


□ 


□ 


□ 


□ 


: a 



How Do You Use Digital Resources in Your Teaching? 



How often do you use digital resources 


in each of these 


ways? 








Al moot all 
the time 


Often 


Sometimes 


Rarely 


Never 


Presented during my lectures/ class 
(e.g., images, audio, etc.). 


□ 


□ 


□ 


□ 


□ 


Posted directly on my course website. 


□ 


□ 


□ 


□ 


□ 


Linked from my course website. 


□ 


□ 


□ 


□ 


□ 
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Almost all 
the time 


Often 


Sometimes 


Rarely 


Never 




Assigned for student research projects Q 

or problem-based learning 
assignments. 


□ 


□ 


□ 


□ 




Assigned to students to create their 1 1 

own digital portfolios and/ or 
multimedia projects. 


□ 


□ 


□ 


□ 




Assigned to students for review and/or 1 1 

study. 


□ 


□ 


□ 


□ 




Used in tests and quizzes. Q 


□ 


□ 


□ 






Presented in my online lectures. Q 


□ 


□ 


□ 






Presented in the context of an online 1 i 

discussion. 


□ 


□ 


□ 


□ ' 




Other. . Q 

Please specify: 


□ 


□ 


□ 


□ 














Motivations for Using Digital Resources 


11 . 


How •would you rate your enthusiasm about using digital resources? 








Lew 

1 2 3 4 5 


6 


High 

7 








□ □ □ □ □ 


□ 


□ 






12. 


How much do you agree or disagree with the following statements about your reasons 
for using digital resources? 






Strongly 

agree 

that this is a 
reason for 
me 


Somewhat 

agree 

that this sa 
reason for 
me 


Somewhat 

disagree 

that th s is a 
reason for 
me 


Strongly 

disagree 

that this is a 
reason for 
me 




I use digital resources in my teaching to provide 
students a context for a topic. 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to get 
students excited about a topic. 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to integrate 
primary source material into the course. 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to integrate 
my research interests into my course. 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to provide 
students with both good and bad examples of 
different kinds of scholarship. 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to teach 

information literacy (i.e., evaluating the online 
materials themselves). 


□ 


□ 


□ 


□ 




I use digital resources in my teaching to teach 
critical thinking skills. 


□ 


□ 


□ 


a 




I use digital resources in my teaching to provide 
students a preview of the course before they 
register. 


□ 


□ 


□ 


□ 
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Motivations for Using Digital Resources 



13. How much do you agree or disagree with the following statements about your reasons 
for using digital resources? 





Strongly 

agree 

that this Is a 
reason for 
me 


Somewhat 

agree 

that this s a 
reason for 
ne 


Somewhat 

disagree 

that th s is a 
reason for 
me 


Strongly 

disagree 

that this is a 
reason for 
rre 


I use digital resources in my teaching because it 
improves my students' learning. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
allows my students to be more creative. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
saves me time. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it is 
more convenient for my students and their 
schedules. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
creates a sense of community for students 
enrolled in my course 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
allows me to do tilings in the classroom that I 
could never do otherwise. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 

provides access to resources that we don't have 
at our college 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because my 
students expect or ask for more technology. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
allows me to stay up-to-date with my 
colleagues. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because the 
administration (deans, chairs, provost) 
encourages me to use digital resources more. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because it 
may help me get promoted or get tenure. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because I like 
or feel very comfortable with the new 
technologies. 


□ 


□ 


□ 


□ 


I use digital resources in my teaching because I 

enjoy having my teaching practices and course 
materials available to anyone in the world who 
would like to use them. 


□ 


□ 


□ 


□ 
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Motivations for NOT Using Digital Resources 1 


These next questions ask about teaching situations where you have decided not to use 
resources. 


digital 


14. 


How much do you agree or disagree with the following statements about your reasons 
for NOT using digital resources in certain situations? 






Strongly 

agree 

that this is a 
reason for 
me 


Somewhat 

agree 

that this sa 
reason for 
me 


Somewhat 

disagree 

that 9i s is a 
reason for 
me 


Strongly 

disagree 

that this is a 
reason for 
me 




I don't use digital resources in certain teaching 
situations, because I don't have time to use 
digital resources. 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 
situations, because they cannot substitute for 
the teaching approaches I use. 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 
situations, because using them distracts from 
the core goals of my teaching 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 
situations, because they are irrelevant to my 
field. 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 
situations, because students don't have the 
information literacy skills to assess the 
credibility of digital resources. 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 
situations, because digital material can be 
presented outside its original context. 


□ 


□ 


□ 


□ 




I don't use digital resources in certain teaching 

situations, because I don't want my students to 
copy or plagiarize material from the web. 


□ 


□ 


□ 


□ 


15. 


How strongly do you agree or disagree with the following statements? 










Strongly 

agree 

that this is a 
reason for 
me 


Somewhat 

agree 

that this sa 
reason for 
me 


Somewhat 

disagree 

that th s is a 
reason for 
me 


Strongly 

disagree 

that this is a 
reason for 
me 




My use of digital resources is very dependent on 
whether they are available to me for free. 


□ 


□ 


□ 


□ 




My use of digital resources is very dependent on 
whether they require registration or a 
password. 


□ 


□ 


□ 


□ 
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Barriers and Frustrations 



This next section deals with barriers or obstacles that make it difficult for you to use digital 
resources the way you would like. 

16. How much do you agree or disagree with the following statements? 





Strongly 

agree 

that 1 is a 
tamer f or 
me 


Somewhat 

agree 

that it is a 

barer for 
me 


Somewhat 

disagree 

that it s a 
barrier ‘or 
me 


Strongly 

disagree 

that t is a 
barrier for 
rre 


I have difficulty using digital resources the way I 
would like, because there are too many 
resources out there for me to take advantage of 
— I am overwhelmed. 


□ 


□ 


□ 


a 


I have difficulty using digital resources the way I 
would like, because I don't know how to locate 
the online materials I need. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because search engines provide 
irrelevant results for my needs. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because the content I need or want 
is just not available online 


□ 


□ 


□ 


a 


I have difficulty using digital resources the way I 
would like, because the academic quality of 
available materials is too poor to meet my 
needs. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because tire digital resources are 
distributed in so many places that it is difficult 
for me to organize them for use in my teaching. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because web sites I would use are 
unreliable, and I can't count on them being 
there when I need them. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't have time to assess 
the credibility of the available resources. 


□ 


□ 


□ 


a 


I have difficulty using digital resources the way I 
would like, because I have difficulty 
understanding the issues surrounding 
copyright and digital collections. 


□ 


□ 


□ 


□ 
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This section deals with technological obstacles that make it difficult for you to use digital 
resources tire way you would like. 

17. How much do you agree or disagree with the following statements? 





Strongly 

agree 

that t is a 
barrier fer 
me 


Somewhat 

agree 

that it is a 

barer for 
ne 


Somewhat 

disagree 

that it s a 
tarrier ‘or 
me 


Strongly 

disagree 

that t is a 
barrrer for 
me 


I have difficulty using digital resources the way I 
would like, because available software is 
unsuitable for viewing and displaying digital 
images. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because available software is 
unsuitable for integrating audio or video into 
my course. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because my students don't have 
reliable access to computers. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because my students don't have a 
high-speed connection. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't have reliable access 
to a computer. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't have reliable access 
to a high-speed connection. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't have reliable access 
to physical resources in my classroom(s) (e g., 
projectors, high-speed connections, etc.). 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because it is difficult to get server 
space or access to a server in order to 
store/host digital resources for teaching. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't have reliable access 
to scanners. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because course management 
software packages (e.g., WebCT, Blackboard) 
are inadequate for my needs. 


□ 


□ 


□ 


□ 


I have difficulty using digital resources the way I 
would like, because I don't know how to save 
presentations to my computer so they can be 
run without a live connection. 


□ 


□ 


□ 


□ 
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Strongly 

agree 

that 1 is a 
barrier for 
me 


Somewhat 

agree 

that it is a 
barer for 
ne 


Somewhat 

disagree 

that it s a 
barrier for 
me 


Strongly 

disagree 

that t is a 
barter for 
rre 


I have difficulty using digital resources the way I 
would like, because web formats (e.g., html or 
pdf) allow me to lmk to whole documents, but 
not to specific excerpts within a text. 


□ 


□ 


□ 


□ 


Other obstacles. 
Please specify: 




□ 


□ 


□ 


□ 




Support and Assistance 


18. How important is it for you to have support or assistance with each of the following 
activities for yorn teaching? 




Support s 
extrenely 
important 


Support is 
very 

inportant 


S.cpcrt is 
soneahat 
important 


S.cpcrt is a 
little 

inportant 


Succor: is 
not at all 
mponant 


Support with finding digital resources. 


□ 


□ 


□ 


□ 


□ 


Support with assessing the credibility of 
digital resources. 


□ 


□ 


□ 


□ 




Support with evaluating the 

appropriateness of resources for 
my teaching goals. 


a 


□ 


□ 


□ 


□ 


Support with interpreting copyright 
laws and/or seeming copyright 
permission. 


□ 


□ 


□ 


□ 


□ 


Support with creating my own website. 


□ 


□ 


□ 


□ 


□ 


Support with importing resources into a 
course website or a database. 


□ 


□ 


□ 


□ 




Support with learning how to use a 

learning management system (e.g., 
WebCT, Blackboard). 


□ 


□ 


□ 


□ 


□ 


Support with integrating resources into 
a learning management system 
(e.g., WebCT, Blackboard). 


a 


□ 


□ 


□ 




Support with digitizing existing 
resources. 


□ 


□ 


□ 


□ 


□ 


Support with gathering, organizing, 
and maintaining digital materials. 


□ 


□ 


□ 


□ 


□ 


Support with trainmg students to find 
or evaluate digital resources. 


□ 


□ 


□ 


□ 


□ 


Support with obtaining or setting up 
technical infrastructure (servers, 
computers, smart classrooms, etc.). 


□ 


□ 


□ 


□ 


□ 


Support with other activities. 
Please specify: 


□ 


□ 


□ 


□ 


□ 
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19. How satisfied or dissatisfied are you with the support you have received from the 
following sources? 





Extremely 
satisfied 
with :hs 
so^ceof 
support 


Somewhat 

satisfied 

wth this 
source of 
sieecrt 


Somewhat 
dissatisfied 
with this 
source of 
support 


Extremely 
dissatisfied 
wthths 
source of 
support 


1 have not 
sou^it 
support frorr 
this source 


Campus educational technology or IT 
support staff 


□ 


□ 


□ 


□ 


□ 


Campus librarians 


□ 


□ 


□ 


□ 


□ 


My departmental or college staff 


□ 


□ 


□ 


a 


□ 


Graduate students 


□ 


□ 


□ 


□ 


□ 


Undergraduate students 


□ 


□ 


□ 


a 


□ 


Other faculty 


□ 


□ 


□ 


□ 


□ 


Friends or family 


□ 


□ 


□ 


a 


□ 


Online help or guides 


□ 


□ 


□ 


□ 


□ 


Workshops 


□ 


□ 


□ 


a 


□ 


Other sources. 
Please specify: 


□ 


□ 


□ 


□ 


□ 



Other Comments 



20. Is there anything else you would like to tell us about your experience with using digital 
resources in undergraduate teaching? 




21. How often do you use each of the tools listed below, for any reason? 





Almost all 
the tine 


Often 


Scmetines 


Rarely 


Never 


A personal computer 


□ 


□ 


□ 


□ 


□ 


The World-Wide Web 


□ 


□ 


□ 


□ 


□ 


Email 


□ 


□ 


□ 


a 


□ 


PowerPoint 


□ 


□ 


□ 


□ 


□ 


An online library catalog 


□ 


□ 


□ 


□ 


□ 


A traditional library card catalog 


□ 


□ 


□ 


□ 


□ 


Abstracting and indexing databases 


□ 


□ 


□ 


a 


□ 
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22. At what institution(s) do you teach? 



(Check all that apply) 

^ Antelope Valiev College 


Q Occidental College 


^ Claremont McKenna College 


Q Oxnard College 


^ College of Marin 


Q Pitzer College 


^ College of the Alameda 


Q Pomona College 


^ Columbia College 


Q Porterville College 


^ East LA College 


Q Saddleback College 


^ Foothill College 


Q San Joaquin Delta College 


^ Golden West College 


Q Santiago Canyon College 


^ Harvey Mudd College 


Q Scripps College 


^ LA Harbor College 


Q UC Berkeley 


^ LA Southwest College 


□ UC Davis 


^ LA Trade Tech College 


Q UC Los Angeles 


^ Mendocino College 


Q UC San Diego 


^1 Mills College 


Q UC Santa Barbara 


^ Mission College 


Q Whittier College 


^ Monterey Peninsula College 


Q Other 


23. What is your job title? 


Please specify: 



CD Instructor 
Q Lecturer 
d Adjunct Professor 
□ Assistant Professor 
Q Associate Professor 
Q Professor 

Q Retired or Emeritus Professor 
Q Graduate student as main instructor 
Q Other. Please specify: 

24. Do you teach part or full-time? 

Q Part-time 
Q Full-time 
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25. What is the highest degree that you have obtained? 

Q Associate's degree 
Q Undergraduate (bachelor's) degree 
Q Master's degree 

□ PhD. 

□ M.D. 

□ J.D. 

□ Other degree. Please specify: 

26. In what year did you receive this degree? 

27. What is your gender? 

Q Male 
Q Female 

28. In what year were you born? 

Thank you very much for your participation in this research study. 

Please return the completed survey in die envelope provided. No postage is necessary, if 
mailed in die United States. 

If you would like to receive a copy of die study's final report or any otiier information, please 
fill out the enclosed postcard and return it witii diis survey. 

Mail to: 

University of California, Berkeley 
Center for Studies in Higher Education 
Digital Collections Research Project 
Soutii Hall Annex, #4650 
Berkeley CA 94720 
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Appendix E: Faculty Survey - Non-response Interview Protocol 

Introduction 



I'm calling for the Center for Studies in Higher Education, UCB, for a study 
about the use of digital resources in college and university teaching. Several 
months ago, we sent you an email inviting you to participate in an online survey 
about your use of digital resources. 



-OR- 

Several months ago, we mailed you a packet with a survey about your use of digital resources. 

According to our records, you didn't respond to that survey. At this time, we'd like to ask you just a few 
questions about your reasons for not responding to the survey. We only need about 5 minutes of your 
time. 

Survey History 

Do you remember receiving that email [or survey]? 

U Yes 

□ No 

To the best of your recollection, which of these reasons might explain why you didn't respond 
to the survey? [Please select all that apply] 

□ You never got the announcement [or the survey]. 

□ You intended to respond, but forgot. 

□ You're too busy or didn't have time. 

□ It wasn't relevant, because you don't teach at a college or university. 

□ It wasn't relevant, because you don't know anything about digital resources. 

□ You are opposed to using digital resources in your teaching. 

I— I Digital resources don't apply to your discipline or to the way that you teach. 

□ You had technical problems with the survey website or with your computer. 

U Other: 

Digital Resource Use 

Now I have a few brief questions about your use of digital resources. 

Please indicate how often you use or have used the following types of digital resources in your 
undergraduate teaching: 
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How often have you used. . . 

Digital images or visual materials (such as drawings, photographs, art, posters, etc.) 

□ Almost all the time 

U Often 

□ Sometimes 

U Rarely 

□ Never 

Online or digitized documents (including translations) 

□ Almost all the time 

U Often 

□ Sometimes 

U Rarely 

□ Never 

Online news or other media sources or archives 

□ Almost all the time 

U Often 

□ Sometimes 

U Rarely 

□ Never 

"Portals" that provide links or URLs relevant to particular disciplinary topics 

□ Almost all the time 

U Often 

□ Sometimes 

U Rarely 

□ Never 

□ Never use digital resources <volunteer> 

On a scale of 1 to 7, where 1 is low enthusiasm and 7 is high enthusiasm , how would you rate 
your enthusiasm about using digital resources? 

1 2 3 4 5 6 7 

DEMOGRAPHICS 

Finally, I have a few questions for statistical purposes. 

In which of these academic discipline(s) do you teach undergraduates? 

(Please select all that apply.) 
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U Languages & Literature: 
o English 

o Languages ; Please specify: 

o Literature ; Please specify: 

o Writing or composition 
U Anthropology 

U Archaeology 

□ Architecture 

□ Art history 

□ Art practice, visual art, other art ; Please specify: 

□ Classics 

U Geography 

U History 

□ Political science 

□ Other; Please specify: 

□ None: I never teach undergraduate courses <volunteer> 

At what institution (or institutions) do you teach? 

What is the highest degree that you have obtained? [select one] 

□ Associate's degree 

U Undergraduate (bachelor's) degree 

□ Master's degree 

□ Ph.D. 

□ M.D. 

□ J.D. 

□ Other degree. 
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In what year did you receive this degree? 

In what year were you born? 

Those are all of the questions I have for you. Do you have any additional thoughts or 
comments for me? 

Thank you for your time, and for answering these questions. 

<Interviewer: Note respondent's gender> 

l_) Male 

□ Female 
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Appendix F: Faculty Survey - Non-response Results Tables 



KEY: 

* p-value, from t-test (estimate of likelihood of difference between the means) 

** p-value, from chi-square analysis (estimate of likelihood of difference between the distributions) 
t Note: Chi-square may not be a valid test - more than 25% of cells have expected counts less than 5. 



Table F.l: Based on non-response survey 





CA/ERALL 


IHv. of Calif. 


Liberal Arts Coll. 


CbtTTTLntyGoll. | 


Question 


Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 






% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


N 


128 


831 




41 


522 




43 


88 




44 


209 




Do you remember receiving 
that email [or survey]? 




























No 


74% (86) 


— 


— 






















Yes 


26% (30) 


— 


— 




















To the best of your 
recollection, which of these 
reasons might explain why 
you didn't respond to the 
survey? Would you say 
that... 


























You never got the 
announcement [or the 
survey]. 


40% (44) 


— 


— 




















You intended to respond, but 
it slipped your mind. 


12% (15) 


— 


— 




















You're too busy or didn't have 
time. 


25% (31) 


— 


— 




















It wasn't relevant, because 
you don’t teach at a college 
or university. 


0% (0) 


— 


— 




















It wasn't relevant, because 
you don't know anything 
about digital resources. 


1% (1) 


— 


— 




















You are opposed to using 
digital resources in your 
teaching. 


0% (0) 


— 


— 
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CA/ERALL 


Uriv. of Calif. 


Liberal Arts Coll. 


Cbrrnxrity Coll. 


Question 


Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 






% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


Digital resources don't apply 
to your discipline or to the 
way that you teach. 


2% (2) 


— 


— 




















You had technical problems 
with the survey website or 
with your computer. 


0% (0) 


— 


— 




















Other reason 


27% (34) 


— 


— 
















































In which of these academic 
discipline(s) do you teach 
undergraduates? 




























None: 1 never teach 
undergraduate courses 


0% (0) 


0% (0) 


— 


0% (0) 


0% (0) 


— 


0% (0) 


0% (0) 


— 


0% (0) 


0% (0) 


— 




Languages & Literature: 


18% (23) 


— 


— 


0% (0) 


— 


— 












— 




English 


18% (23) 


22% (182) 


0.31** 


9.80% (4) 


15% (80) 


0.34“ 


9.30% (4) 


19% (17) 


0.14“ 


34% (15) 


39% (81) 


0.53“ 




Languages 


20% (26) 


18% (149) 


0.52** 


22% (9) 


19% (101) 


0.69“ 


19% (8) 


18% (16) 


0.95“ 


20% (9) 


13% (27) 


0.20“ 




Literature 


29% (37) 


22% (182) 


0.08** 


24% (10) 


23% (121) 


0.86“ 


37% (16) 


20% (18) 


0.04** 


25% (11) 


20% (42) 


0.49“ 




Writing or composition 


27% (34) 


20% (165) 


0.08** 


12% (5) 


13% (67) 


0.91“ 


28% (12) 


17% (15) 


0.15“ 


39% (17) 


38% (79) 


0.95“ 




Anthropology 


5% (6) 


7% (62) 


0.25“ 


7.30% (3) 


7.70% (40) 


0.94“| 


7.00% (3) 


9.10% (8) 


0.68**t 


0% (0) 


6.30% (13) 


0.09“| 




Archaeology 


2% (2) 


3% (24) 


0.39**| 


4.90% (2) 


3.50% (18) 


0.63“| 


0% (0) 


1.10% (1) 


0.48“| 


0% (0) 


1.50% (3) 


0.42**t 




Architecture 


1% (1) 


2% (20) 


0.24“| 


2.40% (1) 


2.10% (11) 


0.89“| 


0% (0) 


1.10% (1) 


0.48**t 


0% (0) 


3.90% (8) 


0.1 9“f 




Art history 


9% (11) 


6% (47) 


0.20** 


7.30% (3) 


4.80% (25) 


0.47“| 


9.30% (4) 


10% (9) 


0.87**| 


9.10% (4) 


6.30% (13) 


0.50“| 




Art practice, visual art, 
other art 


13% (17) 


8% (63) 


0.03“ 


4.90% (2) 


6.70% (35) 


0.65“| 


12% (5) 


6.80% (6) 


0.35“| 


23% (10) 


11% (22) 


0.03“ 




Classics 


5% (6) 


2% (19) 


0.1 1“| 


2.40% (1) 


3.10% (16) 


0.82“| 


9.30% (4) 


1.10% (1) 


0.02“| 


2.30% (1) 


0.97% (2) 


0.47“| 




Geography 


1% (1) 


3% (24) 


0.1 6“| 


0% (0) 


2.90% (15) 


0.27“| 


0% (0) 


0% (0) 


— 


2.30% (1) 


3.90% (8) 


0.6 1“| 




History 


19% (24) 


15% (128) 


0.34“ 


32% (13) 


16% (81) 


0.007“ 


21% (9) 


22% (19) 


0.93 


4.60% (2) 


13% (27) 


0.11“ 




Political science 


14% (18) 


11% (92) 


0.33“ 


15% (6) 


12% (61) 


0.57“| 


16% (7) 


20% (18) 


0.57 


11% (5) 


6.30% (13) 


0.24**t 




Other 


24% (31) 


19% (157) 


0.16“ 


29% (12) 


18% (96) 


0.09“ 


23% (10) 


16% (14) 


0.31 


20% (9) 


20% (42) 


0.98“ 


What is the highest degree 
that you have obtained? 


























□ 


Associate's degree 


1% (1) 


1% (1) 


0.16* 


0% (0) 


0% (0) 


0.73* 


0% (0) 


0% (0) 


0.93* 


2.30% (1) 


0.54% (1) 


0.63* 
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CA/ERALL 


Uriv. of Calif. 


Liberal Arts Coll. 


Cbrrnxrity Coll. 


Question 


Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 






% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 




Undergraduate 
(bachelor's) degree 


3% (4) 


2% (18) 




2.40% (1) 


1.70% (8) 




0% (0) 


1.20% (1) 




6.80% (3) 


4.90% (9) 






Master's degree 


30% (38) 


26% (198) 


12% (5) 


11% (54) 


12% (5) 


9.80% (8) 


64% (28) 


71% (132) 




Ph.D. 


63% (81) 


69% (519) 


83% (34) 


86% (406) 


88% (38) 


88% (72) 


20% (9) 


19% (36) 




M.D. 


2% (2) 


1 % (4) 


2.40% (1) 


0.21% (1) 


0% (0) 


1.20% (1) 


2.30% (1) 


1.10% (2) 




J.D. 


0% (0) 


1 % (7) 


0% (0) 


0.84% (4) 


0% (0) 


0% (0) 


0% (0) 


1 .60% (3) 




Other degree 


2% (2) 


1 % (3) 


0% (0) 


0.21% (1) 


0% (0) 


0% (0) 


4.60% (2) 


1.10% (2) 


In what year did you receive 
this degree? (mean ± SD) 


1985 ±11 


1986 ±12 


0.69* 


1985 ± 11 


1986 ±11 


0.73* 


1985 ±11 


1989 ±11 


0.07* 


1986 ± 12 


1986 ±12 


0.68* 


In what year were you born? 
(mean ± SD) 


1953 ±10 


1954 ±11 


0.49* 


1953 ± 11 


1954 ±11 


0.44* 


1952 ±11 


1957 ±11 


0.04* 


1954 ±9.4 


1952 ±11 


0.22* 


Gender 




























Male 


57% (73) 


49% (362) 


0.09** 


68% (28) 


55% (257) 


0.10** 


51% (22) 


48% (38) 


0.70** 


52% (23) 


35% (64) 


0.03** 




Female 


43% (55) 


51% (379) 


32% (13) 


45% (212) 


49% (21) 


53% (42) 


48% (21) 


65% (119) 


How often do you or have 
you used the following types 
of digital resources in your 
undergraduate teaching: 


























Digital images or visual 
materials 




























Almost all the time 


23% (30) 


28% (227) 


0.08* 


27% (11) 


27% (140) 


0.87* 


16% (7) 


24% (20) 


0.06* 


27% (12) 


32% (63) 


0.19* 




Often 


19% (24) 


24% (192) 


15% (6) 


22% (115) 


19% (8) 


26% (22) 


23% (10) 


27% (54) 




Sometimes 


23% (30) 


23% (183) 


27% (11) 


23% (116) 


23% (10) 


22% (19) 


20% (9) 


22% (44) 




Rarely 


29% (24) 


14% (111) 


20% (8) 


13% (68) 


19% (8) 


18% (15) 


18% (8) 


13% (26) 




Never 


15% (19) 


12% (95) 


9.80% (4) 


14% (73) 


23% (10) 


11% (9) 


11% (5) 


6.50% (13) 


Online or digitized documents 
(including translations) 




























Almost all the time 


9% (11) 


11% (85) 


0.05* 


5% (2) 


11% (54) 


0.42* 


12% (5) 


16% (13) 


0.67* 


9.10% (4) 


9.00% (17) 


0.18* 




Often 


26% (33) 


18% (137) 


25% (10) 


18% (89) 


23% (10) 


23% (19) 


30% (13) 


15% (28) 




Sometimes 


25% (32) 


22% (269) 


30% (12) 


22% (111) 


30% (13) 


22% (18) 


16% (7) 


21% (40) 




Rarely 


18% (23) 


15% (117) 


20% (8) 


15% (73) 


19% (8) 


7.30% (6) 


16% (7) 


20% (37) 




Never 


22% (28) 


35% (269) 


20% (8) 


34% (169) 


16% (7) 


32% (26) 


30% (13) 


35% (67) 
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CA/ERALL 


Uiv. of Calif. 


Liberal Arts Coll. 


Cbrrnxrity Coll. 


Question 


Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 




Non- 

response 

follow-up 


Faculty 

Survey 






% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


Online news or other media 
sources or archives 




























Almost all the time 


9% (11) 


9% (74) 


0.31* 


10% (4) 


8.90% (44) 


0.88* 


7.00% (3) 


14% (12) 


0.02* 


9.10% (4) 


8.60% (17) 


0.04* 




Often 


20% (26) 


26% (204) 


18% (7) 


20% (100) 


21% (9) 


39% (33) 


23% (10) 


35% (68) 




Sometimes 


27% (34) 


29% (228) 


23% (9) 


26% (130) 


26% (11) 


20% (17) 


32% (14) 


39% (76) 




Rarely 


24% (31) 


13% (106) 


30% (12) 


16% (80) 


26% (11) 


9.50% (8) 


18% (8) 


8.60% (17) 




Never 


20% (25) 


22% (177) 


20% (8) 


29% (143) 


21% (9) 


17% (14) 


18% (8) 


9.60% (19) 


Portals that provide links or 
URLs relevant to particular 
disciplinary topics 




























Almost all the time 


10% (13) 


12% (96) 


0.49* 


18% (7) 


12% (60) 


0.60* 


9.30% (4) 


12% (10) 


0.42* 


4.60% (2) 


12% (24) 


0.17* 




Often 


19% (24) 


23% (179) 


20% (8) 


21% (102) 


19% (8) 


27% (23) 


18% (8) 


26% (51) 




Sometimes 


33% (42) 


28% (221) 


23% (9) 


28% (139) 


35% (15) 


29% (25) 


41% (18) 


28% (54) 




Rarely 


15% (19) 


14% (112) 


18% (7) 


16% (75) 


14% (6) 


9.40% (8) 


14% (6) 


13% (26) 




Never 


23% (29) 


23% (179) 


23% (9) 


24% (121) 


23% (10) 


22% (19) 


23% (10) 


20% (39) 


Total Use 


7.5 ±3.7 


7.5 ±3.8 


0.92* 


7.6 ±3.6 


7.1 ±3.9 


0.45* 


7.3 ±3.6 


8.2 ±3.9 


0.18* 


7.6 ±4.0 


7.9 ±3.7 


0.62* 


On a scale of 1 to 7, where 1 
is low enthusiasm and 7 is 
high enthusiasm, how would 
you rate your enthusiasm 
about using digital 
resources? (mean ± SD) 


4.9 ±1.7 


4.9 ±1.7 


0.60* 


4.9 ±1.9 


4.8 ±1.8 


0.74* 


5.1 ±1.7 


4.7 ±1.6 


0.28* 


4.8 ±1.7 


5.0 ±1.7 


0.50* 
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Table F.2: Based on all eligible subjects 





CA/ERALL 


liniv. of Calif. 


Liberal Arts Coll. 


Cbrrvrxinity Coll. 


Question 


All non- 
responders 


Faculty 

Survey 




All non- 
responders 


Faculty 

Survey 




All non- 
respond- 
ers 


Faculty 

Survey 




All non- 
responders 


Faculty 

Survey 






% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


% (N) 


% (N) 


p-val 


N 


3,612 


831 




1,894 


522 




340 


88 




1,373 


209 




Discipline: 




























English language or lit 


27% (947) 


27% (216) 




19% (358) 


20% (104) 




22% (74) 


22% (19) 




38% (515) 


45% (93) 






Foreign language or lit 


19% (669) 


21% (169) 




22% (408) 


26% (134) 




20% (68) 


17% (15) 




14% (193) 


9.6% (20) 






Anthropology 


6% (216) 


7% (58) 




7.7% (146) 


7.7% (40) 




6.50% 

(22) 


8.00% (7) 




3.5% (48) 


5.3% (11) 






Archaeology 


0.10% (4) 


0.40% (3) 




0.2% (4) 


0.6% (3) 




0% (0) 


0% (0) 




0% (0) 


0% (0) 






Architecture 


4% (120) 


3% (24) 




5.0% (94) 


3.3% (17) 




0% (0) 


0% (0) 




1.9% (26) 


3.4% (7) 






Art history 


3% (99) 


3% (28) 


0.004** 

[0.01** 

lumped] 


2.8% (52) 


3.0% (18) 


0.05“ 

[0.03** 

lumped] 


5.00% 

(17) 


5.70% (5) 


0.92“ 
[0.86** 
lumped] | 


2.2% (30) 


2.4% (5) 


0.13“ 

[0.24** 

lumped] 




Art practice or applied 
arts 


9% (320) 


6% (45) 


6.0% (114) 


4.0% (21) 


5.60% 

(19) 


4.60% (4) 


14% (187) 


9.6% (20) 




Geography 


3% (116) 


3% (21) 




4.1% (77) 


2.7% (14) 




0.29% (1) 


0% (0) 




2.80% (38) 


3.4% (7) 






History 


14% (509) 


14% (112) 




18% (337) 


14% (73) 




15% (50) 


20% (18) 




8.9% (122) 


10% (21) 






Political Science 


9% (323) 


12% (96) 




10% (191) 


13% (68) 




18% (62) 


19% (17) 




5.1% (70) 


5.3% (11) 






Classics 


1 % (47) 


2% (13) 




2.1% (40) 


2.3% (12) 




1.80% (6) 


1.10% (1) 




0.07% (1) 


0% (0) 






Languages and lit, n.s. 


2% (67) 


1% (10) 




0.16% (3) 


0% (0) 




2.10% (7) 


0% (0) 




4.2% (57) 


4.8% (10) 






Art and architecture, 
n.s. 


3% (113) 


2% (14) 




1.90% (36) 


1.70% (9) 




2.70% (9) 


2.30% (2) 




5.0% (68) 


1.40% (3) 




Gender 




























Male 


56% (1615) 


51% (359) 


0.01** 


61% (937) 


55% (247) 


0.02“ 


52% (149) 


47% (39) 


0.38“ 


50% (529) 


41% (73) 


0.03“ 




Female 


44% (1258) 


49% (347) 


39% (590) 


45% (199) 


48% (135) 


53% (44) 


50% (533) 


59% (104) 
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Appendix G: Principal Component Analysis - Factor Matrices 

Table G.l: Types (and sources) of digital resources (5 factors) 





Use Factor 1 


Use Factor 2 


Use Factor 3 


Use Factor 4 


Use Factor 5 


General-purpose 
and reference 
materials 


Images and 
audiovisual 
materials 


Historical 

documents, maps, 
and primary 
sources 


Data, 

news/media, and 
governmental 
resources 


Discussion and 
curricular materials 


q05a_images 


5a. How often do you use... Images 
or visual materials 


-0.05 


0.18 


0.20 


-0.05 


-0.15 


q05b_maps 


5b. How often do you use... Maps 


-0.15 


0.04 


0.31 


0.13 


-0.10 


q05c_simul 


5c. How often do you use... 
Simulations or animations 


-0.17 


0.23 


-0.02 


0.04 


0.22 


q05d_film 


5d. How often do you use... Digital 
film or video 


-0.07 


0.28 


-0.06 


0.06 


-0.06 


q05e_audio 


5e. How often do you use... Audio 
materials 


-0.02 


0.22 


-0.07 


0.02 


0.00 


q05f_manusc 


5f. How often do you use... Digital 
facsimiles of ancient or historical 
manuscripts 


-0.08 


-0.08 


0.34 


-0.03 


0.05 


q05g_docs 


5g. How often do you use... Online or 
digitized documents 


0.03 


-0.13 


0.16 


0.02 


0.16 


q05h_govdocs 


5h. How often do you use... 
Government documents 


-0.09 


-0.10 


0.07 


0.38 


0.05 


q05i_data 


5i. How often do you use... Data 
archives 


-0.11 


-0.04 


0.02 


0.41 


-0.01 


q05j_news 


5j. How often do you use... News or 
other media sources and archives 


0.07 


0.07 


-0.15 


0.27 


-0.09 


q05k_refer 


5k. How often do you use... Online 
reference resources 


0.19 


0.02 


-0.08 


-0.03 


-0.04 


q051_portal 


51. How often do you use... “Portals” 
that provide links or URL's relevant 
to particular disciplinary topics 


0.21 


0.02 


-0.09 


-0.06 


0.01 


q05m_blog 


5m. How often do you use... 
Personal online diaries 


-0.11 


0.14 


-0.19 


0.07 


0.38 


q05n_discuss 


5n. How often do you use... Online 
class discussions 


-0.05 


-0.05 


-0.02 


-0.07 


0.49 
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Use Factor 1 


Use Factor 2 


Use Factor 3 


Use Factor 4 


Use Factor 5 


General-purpose 
and reference 
materials 


Images and 
audiovisual 
materials 


Historical 

documents, maps, 
and primary 
sources 


Data, 

news/media, and 
governmental 
resources 


Discussion and 
curricular materials 


q05o_curric 


5o. How often do you use... 
Curricular materials and websites 
that are created by other faculty 
and/or other institutions 


0.08 


0.08 


-0.15 


-0.04 


0.17 


q05p_coursepack 


5p. How often do you use... Digital 
readers or coursepacks 


-0.09 


-0.08 


0.07 


-0.02 


0.42 


q06a_searcheng 


6a. How often do you use digital 
resources from... Search 
engines/directories 


0.19 


0.06 


-0.09 


0.02 


-0.14 


q06b_owncoll 


6b. How often do you use digital 
resources from... My own personal 
collection of digital materials 


-0.02 


0.07 


0.15 


-0.06 


0.04 


q06c_free_img 


6c. How often do you use digital 
resources from... Public online image 
databases 


0.05 


0.06 


0.17 


-0.07 


-0.06 


q06d_comm_img 


6d. How often do you use digital 
resources from... Commercial image 
databases 


0.06 


0.21 


-0.04 


-0.10 


-0.09 


q06e_campus_img 


6e. How often do you use digital 
resources from... Campus image 
databases from my own institution 


0.08 


0.14 


0.03 


-0.17 


-0.03 


q06f_portal 


6f. How often do you use digital 
resources from... “Portals” that 
provide links or URL's relevant to 
particular disciplinary topics 


0.25 


-0.02 


-0.05 


-0.11 


-0.04 


q06g_exhibit 


6g. How often do you use digital 
resources from... Online exhibits 


0.12 


0.07 


0.09 


-0.13 


-0.04 


q06h_diglib 


6h. How often do you use digital 
resources from... Library collections 


0.17 


-0.18 


0.13 


-0.04 


0.01 


q06i_journal 


6i. How often do you use digital 
resources from... Online journals 


0.16 


-0.24 


0.17 


0.05 


-0.03 


q06j_media 


6j. How often do you use digital 
resources from... Media sites 


0.17 


-0.01 


-0.15 


0.21 


-0.14 
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Table G.2: Ways that people use resources (4 factors) 





How Factor 
1 


How Factor 
2 


How Factor 
3 


How Factor 
4 


Student 

assignments 


Web posting 


Online courses 


In-class 

presentation 


q 1 Oa_in_class 


10a. How often do you use digital 
resources... Presented during my 
lectures/class 


-0.15 


-0.06 


-0.13 


0.95 


q 1 0b_post_web 


10b. How often do you use digital 
resources... Posted directly on my 
course website. 


-0.17 


0.61 


-0.08 


-0.03 


q 1 Oc_link_web 


10c. How often do you use digital 
resources... Linked from my 
course website. 


-0.11 


0.57 


-0.09 


-0.04 


qlOd_st_project 


1 0d. How often do you use digital 
resources... Assigned for student 
research projects or problem- 
based learning assignments. 


0.55 


-0.14 


-0.10 


-0.12 


q 1 Oe_st_portfolio 


lOe. How often do you use digital 
resources... Assigned to students 
to create their own digital portfolios 
and/or multimedia projects. 


0.47 


-0.20 


-0.05 


0.04 


qlOf_study 


1 0f. How often do you use digital 
resources... Assigned to students 
for review and/or study. 


0.39 


0.09 


-0.07 


-0.21 


q 1 0g_test 


1 0g. How often do you use digital 
resources... Used in tests and 
quizzes. 


0.10 


-0.08 


0.09 


0.31 


q 1 Oh_online_lec 


1 0h. How often do you use digital 
resources... Presented in my 
online lectures. 


-0.14 


-0.08 


0.58 


0.02 


q 1 Oi_online_disc 


1 0i. How often do you use digital 
resources... Presented in the 
context of an online discussion. 


-0.04 


-0.10 


0.62 


-0.19 
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Table G.3: Reasons for using or not using resources (8 factors) 





Reason 
Factor 1 


Reason 

Factor2 


Reason 

Factor3 


Reason 

Factor4 


Reason 

Factor5 


Reason 

Factor6 


Reason 

Factor7 


Reason 

Factor8 


Pedagogical 

reasons 


Expectations 
& reputation 


Inappropri- 

ateness 


Concerns 

about 

students' 

interpretation 

and 

information 

literacy 


Time, 

convenience, 

access 


Teaching 
information 
literacy 
and critical 
thinking 


Making 

information 

publicly 

available 


Using 
free and 
publicly 
available 
materials 


q 1 2a_why_context 


12a. Agree/Disagree: 1 use digital resources in my 
teaching to provide students a context for a topic. 


0.24 


-0.02 


0.07 


-0.02 


-0.15 


-0.13 


0.08 


0.05 


q 1 2b_why_excited 


12b. Agree/Disagree: 1 use digital resources in my 
teaching to get students excited about a topic. 


0.24 


0.02 


0.04 


0.01 


-0.15 


-0.12 


0.01 


0.03 


q 1 2c_why_primsource 


12c. Agree/Disagree: 1 use digital resources in my 
teaching to integrate primary source material into 
the course. 


0.11 


-0.16 


0.05 


0.00 


-0.06 


0.09 


0.17 


0.13 


q 1 2d_why_research 


1 2d. Agree/Disagree: 1 use digital resources in my 
teaching to integrate my research interests into my 
course. 


0.02 


-0.11 


0.05 


0.00 


-0.08 


0.05 


0.36 


0.14 


q 1 2e_why_examples 


12e. Agree/Disagree: 1 use digital resources in my 
teaching to provide students with both good and 
bad examples of different kinds of scholarship. 


-0.15 


-0.08 


-0.01 


-0.05 


0.03 


0.54 


0.02 


0.03 


q 1 2f_why_infolit 


12f. Agree/Disagree: 1 use digital resources in my 
teaching to teach information literacy (i.e., 
evaluating the online materials themselves). 


-0.06 


0.02 


-0.01 


-0.01 


-0.03 


0.53 


-0.19 


-0.09 


q 1 2g_why_critthink 


12g. Agree/Disagree: 1 use digital resources in my 
teaching to teach critical thinking skills. 


0.06 


0.01 


0.01 


-0.03 


-0.17 


0.30 


-0.01 


-0.05 


q 1 2h_why_preview 


12h. Agree/Disagree: 1 use digital resources in my 
teaching to provide students a preview of the 
course before they register. 


-0.07 


0.12 


0.01 


0.01 


-0.01 


-0.16 


0.41 


-0.03 


q 1 3a_why_learn 


13a. Agree/Disagree: 1 use digital resources in my 
teaching because it improves my students' 
learning. 


0.22 


0.01 


0.02 


-0.04 


0.01 


-0.12 


-0.10 


-0.06 
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Reason 
Factor 1 


Reason 

Factor2 


Reason 

Factor3 


Reason 

Factor4 


Reason 

Factor5 


Reason 

Factor6 


Reason 

Factor7 


Reason 

Factor8 


Pedagogical 

reasons 


Expectations 
& reputation 


Inappropri- 

ateness 


Concerns 

about 

students' 

interpretation 

and 

information 

literacy 


Time, 

convenience, 

access 


Teaching 
information 
literacy 
and critical 
thinking 


Making 

information 

publicly 

available 


Using 
free and 
publicly 
available 
materials 


q 1 3b_why_creative 


13b. Agree/Disagree: 1 use digital resources in my 
teaching because it allows my students to be more 
creative. 


0.18 


0.01 


-0.04 


0.06 


-0.05 


0.06 


-0.11 


-0.13 


q 1 3c_why_savetime 


13c. Agree/Disagree: 1 use digital resources in my 
teaching because it saves me time. 


-0.09 


-0.09 


-0.03 


-0.02 


0.46 


-0.02 


0.02 


0.09 


q 1 3d_why_convenient 


13d. Agree/Disagree: 1 use digital resources in my 
teaching because it is more convenient for my 
students and their schedules. 


-0.05 


0.03 


0.07 


-0.04 


0.45 


-0.06 


-0.06 


-0.03 


q 1 3e_why_community 


13e. Agree/Disagree: 1 use digital resources in my 
teaching because it creates a sense of community 
for students enrolled in my course. 


-0.03 


0.09 


0.08 


-0.06 


0.08 


-0.02 


0.21 


-0.03 


q 1 3f_why_domore 


13f. Agree/Disagree: 1 use digital resources in my 
teaching because it allows me to do things in the 
classroom that 1 could never do otherwise. 


0.21 


0.00 


0.03 


0.01 


0.09 


-0.10 


-0.15 


-0.07 


q 1 3g_why_access 


13g. Agree/Disagree: 1 use digital resources in my 
teaching because it provides access to resources 
that we don't have at our college. 


0.15 


-0.18 


-0.04 


0.16 


0.28 


0.02 


-0.16 


0.02 


q 1 3h_why_studexpect 


13h. Agree/Disagree: 1 use digital resources in my 
teaching because my students expect or ask for 
more technology. 


0.06 


0.31 


-0.04 


0.03 


0.00 


-0.09 


-0.18 


-0.04 


q 1 3i_why_uptodate 


13i. Agree/Disagree: 1 use digital resources in my 
teaching because it allows me to stay up-to-date 
with my colleagues. 


0.03 


0.22 


-0.05 


0.07 


0.08 


-0.01 


-0.12 


-0.04 


q 1 3j_why_admin 


13j. Agree/Disagree: 1 use digital resources in my 
teaching because the administration encourages 
me to use digital resources more. 


-0.06 


0.36 


0.01 


-0.02 


-0.08 


-0.06 


0.03 


0.03 



http:/ / digitalresourcestudy.berkeley.edu 



10-35 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 





Reason 
Factor 1 


Reason 

Factor2 


Reason 

Factor3 


Reason 

Factor4 


Reason 

Factor5 


Reason 

Factor6 


Reason 

Factor7 


Reason 

Factor8 


Pedagogical 

reasons 


Expectations 
& reputation 


Inappropri- 

ateness 


Concerns 

about 

students' 

interpretation 

and 

information 

literacy 


Time, 

convenience, 

access 


Teaching 
information 
literacy 
and critical 
thinking 


Making 

information 

publicly 

available 


Using 
free and 
publicly 
available 
materials 


q 1 3k_why_tenure 


13k. Agree/Disagree: 1 use digital resources in my 
teaching because it may help me get promoted or 
get tenure. 


-0.11 


0.35 


0.06 


-0.09 


-0.15 


-0.01 


0.12 


0.08 


ql31_why_like 


131. Agree/Disagree: 1 use digital resources in my 
teaching because 1 like or feel very comfortable 
with the new technologies. 


0.14 


-0.06 


-0.03 


0.05 


0.14 


-0.11 


0.03 


-0.03 


q 1 3m_why_avail_world 


13m. Agree/Disagree: 1 use digital resources in my 
teaching because 1 enjoy having my teaching 
practices and course materials available to anyone 
in the world who would like to use them. 


-0.03 


-0.02 


0.02 


-0.01 


0.04 


-0.01 


0.32 


-0.08 


q 1 4a_whynot_notime 


14a. Agree/Disagree: 1 don't use digital resources , 
because 1 don't have time to use digital resources. 


0.01 


0.11 


0.30 


-0.15 


0.11 


0.06 


-0.30 


0.03 


q 1 4b_whynot_nosubst 


14b. Agree/Disagree: 1 don't use digital resources , 
because they cannot substitute for the teaching 
approaches 1 use. 


0.05 


0.00 


0.41 


-0.08 


0.09 


-0.02 


-0.07 


-0.08 


q 1 4c_whynot_distract 


14c. Agree/Disagree: 1 don't use digital resources , 
because using them distracts from the core goals of 
my teaching. 


0.01 


0.02 


0.39 


-0.06 


-0.07 


-0.02 


0.15 


0.01 


q 1 4d_whynot_irrelevant 


14d. Agree/Disagree: 1 don't use digital resources , 
because they are irrelevant to my field. 


0.04 


-0.05 


0.30 


0.01 


-0.05 


-0.02 


0.11 


-0.04 


q 1 4e_whynot_infolit 


14e. Agree/Disagree: 1 don't use digital resources , 
because students don't have the information 
literacy skills to assess the credibility of digital 
resources. 


0.04 


-0.04 


-0.05 


0.38 


-0.03 


0.03 


-0.04 


0.05 


q 1 4f_whynot_decontext 


14f. Agree/Disagree: 1 don’t use digital resources , 
because digital material can be presented outside 
its original context. 


0.04 


-0.06 


-0.06 


0.43 


0.03 


-0.03 


0.04 


-0.09 
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Reason 
Factor 1 


Reason 

Factor2 


Reason 

Factor3 


Reason 

Factor4 


Reason 

Factor5 


Reason 

Factor6 


Reason 

Factor7 


Reason 

Factor8 


Pedagogical 

reasons 


Expectations 
& reputation 


Inappropri- 

ateness 


Concerns 

about 

students' 

interpretation 

and 

information 

literacy 


Time, 

convenience, 

access 


Teaching 
information 
literacy 
and critical 
thinking 


Making 

information 

publicly 

available 


Using 
free and 
publicly 
available 
materials 


q 1 4g_whynot_plagiarize 


14g. Agree/Disagree: 1 don't use digital resources , 
because 1 don't want my students to copy or 
plagiarize material from the web. 


0.02 


0.05 


-0.09 


0.43 


-0.01 


-0.13 


0.01 


0.02 


q 1 5 a_depend_free 


15a. Agree/Disagree: My use of digital resources is 
very dependent on whether they are available to 
me for free. 


-0.03 


-0.01 


0.02 


-0.07 


0.08 


0.00 


-0.03 


0.52 


q 1 5b_depend_password 


15b. Agree/Disagree: My use of digital resources is 
very dependent on whether they require registration 
or a password. 


-0.04 


0.04 


-0.07 


0.06 


-0.04 


-0.09 


0.07 


0.55 
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Table G.4: Barriers to digital resource use (5 factors) 





Barrier 
Factor 1 


Barrier 

Factor2 


Barrier 

Factor3 


Barrier 

Factor4 


Barrier 

Factor5 


Finding resources 


High-end 
multimedia 
equipment and 
software 


Personal 
equipment t 


Resource 
availability or 
existence 


Student 

equipment 


q 1 6a_barrier_overwhelm 


16a. Agree/Disagree: 1 have difficulty using 
digital resources because there are too 
many resources out there for me to take 
advantage of— 1 am overwhelmed. 


0.27 


-0.13 


0.03 


-0.10 


-0.02 


q 1 6b_barrier_locate 


16b. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don't know how 
to locate the online materials 1 need. 


0.29 


-0.08 


-0.05 


-0.12 


0.00 


q 1 6c_barrier_search_eng 


16c. Agree/Disagree: 1 have difficulty using 
digital resources because search engines 
provide irrelevant results for my needs. 


0.16 


-0.09 


-0.04 


0.12 


0.03 


q 1 6d_barrier_unavail 


16d. Agree/Disagree: 1 have difficulty using 
digital resources because the content 1 
need or want is just not available online. 


-0.11 


-0.01 


0.00 


0.49 


-0.06 


q 1 6e_barrier_quality 


16e. Agree/Disagree: 1 have difficulty using 
digital resources because the academic 
quality of available materials is too poor to 
meet my needs. 


-0.08 


-0.11 


0.07 


0.50 


-0.03 


q 1 6f_barrier_distrib 


16f. Agree/Disagree: 1 have difficulty using 
digital resources because the digital 
resources are distributed in so many 
places that it is difficult for me to organize 
them for use in my teaching. 


0.20 


-0.06 


-0.05 


0.03 


0.01 


q 1 6g_barrier_unreliable 


16g. Agree/Disagree: 1 have difficulty using 
digital resources because web sites 1 would 
use are unreliable, and 1 can’t count on 
them being there when 1 need them. 


0.06 


0.05 


-0.13 


0.18 


0.05 


q 1 6h_barrier_credibility 


16h. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don’t have time 
to assess the credibility of the available 
resources. 


0.19 


-0.15 


0.06 


0.09 


-0.03 
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Barrier 
Factor 1 


Barrier 

Factor2 


Barrier 

Factor3 


Barrier 

Factor4 


Barrier 

Factor5 






Finding resources 


High-end 
multimedia 
equipment and 
software 


Personal 
equipment t 


Resource 
availability or 
existence 


Student 

equipment 


q 1 6i_barrier_copyright 


16i. Agree/Disagree: 1 have difficulty using 
digital resources because 1 have difficulty 
understanding the issues surrounding 
copyright and digital collections. 


0.20 


-0.02 


-0.02 


-0.07 


-0.02 


q 1 7 a_barrier_software_img 


17a. Agree/Disagree: 1 have difficulty using 
digital resources because available 
software is unsuitable for viewing and 
displaying digital images. 


-0.05 


0.35 


-0.15 


-0.01 


0.01 


q 1 7b_barrier_software_audio_v 

ide 


17b. Agree/Disagree: 1 have difficulty using 
digital resources because available 
software is unsuitable for integrating audio 
or video into my course. 


-0.03 


0.37 


-0.19 


-0.05 


0.03 


q 1 7 c_barrier_stud_computer 


17c. Agree/Disagree: 1 have difficulty using 
digital resources because my students 
don't have reliable access to computers. 


-0.02 


-0.06 


-0.06 


-0.04 


0.54 


q 1 7d_barrier_stud_connect 


17d. Agree/Disagree: 1 have difficulty using 
digital resources because my students 
don't have a high-speed connection. 


-0.02 


-0.07 


-0.06 


-0.01 


0.55 


q 1 7 e_barrier_me_computer 


17e. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don't have 
reliable access to a computer. 


-0.04 


-0.18 


0.48 


0.04 


-0.04 


q 1 7f_barrier_me_connect 


17f. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don't have 
reliable access to a high-speed connection. 


0.01 


-0.21 


0.48 


0.01 


-0.05 


q 1 7g_barrier_classroom_equip 


17g. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don't have 
reliable access to physical resources in my 
classroom(s). 


-0.12 


0.14 


0.16 


0.00 


0.02 


q 1 7h_barrier_server 


17h. Agree/Disagree: 1 have difficulty using 
digital resources because it is difficult to 
get server space or access to a server in 
order to store/host digital resources for 
teaching. 


-0.06 


0.14 


0.16 


-0.02 


-0.04 
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Barrier 
Factor 1 


Barrier 

Factor2 


Barrier 

Factor3 


Barrier 

Factor4 


Barrier 

Factor5 






Finding resources 


High-end 
multimedia 
equipment and 
software 


Personal 
equipment t 


Resource 
availability or 
existence 


Student 

equipment 


q 17i_barrier_scanner 


17i. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don't have 
reliable access to scanners. 


-0.04 


0.14 


0.21 


-0.03 


-0.13 


q 1 7j_barrier_lms 


17j. Agree/Disagree: 1 have difficulty using 
digital resources because course 
management software packages are 
inadequate for my needs. 


-0.15 


0.36 


-0.06 


0.08 


-0.11 


q 1 7k_barrier_save_pres 


17k. Agree/Disagree: 1 have difficulty using 
digital resources because 1 don’t know how 
to save presentations to my computer so 
they can be run without a live connection. 


0.17 


0.12 


0.03 


-0.21 


-0.11 


q 1 71_barrier_link_excerpt 


171. Agree/Disagree: 1 have difficulty using 
digital resources because web formats 
allow me to link to whole documents, but 
not to specific excerpts within a text. 


0.08 


0.23 


-0.08 


-0.16 


0.00 
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Table G.5: Activities that require support (2 factors) 





Support Factorl 


Support Factor2 


Technical activities 


Intellectual and content- 
based activities 


q 1 8a_support_find 


18a. How important is it for you to have 
support with... finding digital resources. 


-0.07 


0.28 


q 1 8b_support_credib 


18b. How important is it for you to have 
support with... assessing the credibility of 
digital resources. 


-0.16 


0.38 


q 1 8c_support_appropr 


18c. How important is it for you to have 
support with... evaluating the appropriateness 
of resources for my teaching goals. 


-0.16 


0.38 


q 1 8d_support_copyright 


18d. How important is it for you to have 
support with... interpreting copyright laws 
and/or securing copyright permission. 


-0.05 


0.23 


q 1 8e_support_website 


18e. How important is it for you to have 
support with... creating my own website. 


0.19 


-0.08 


q 1 8f_support_import 


18f. How important is it for you to have 
support with... importing resources into a 
course website or a database. 


0.20 


-0.07 


q 1 8g_support_learn_lms 


18g. How important is it for you to have 
support with... learning howto use a learning 
management system. 


0.21 


-0.09 


q 1 8h_support_integrate_lms 


18h. How important is it for you to have 
support with... integrating resources into a 
learning management system. 


0.21 


-0.09 


q 1 8i_support_digitize 


18i. How important is it for you to have 
support with... digitizing existing resources. 


0.20 


-0.08 


q 1 8j_support_organize 


18j. How important is it for you to have 
support with... gathering, organizing, and 
maintaining digital materials. 


0.14 


0.01 


q 1 8k_support_train 


18k. How important is it for you to have 
support with... training students to find or 
evaluate digital resources. 


0.05 


0.13 


q 1 81_support_infrastr 


181. How important is it for you to have 
support with... obtaining or setting up 
technical infrastructure. 


0.17 


-0.06 
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Appendix H: Faculty Survey - Results Tables 



Table H.l: Faculty survey responses, broken out by institution type 





All 


University 

of 

California 


Liberal 

arts 

colleges 


Community 

colleges 


N 


831 


522 


90 


206 


How often do you use the following types of digital resources in 
your undergraduate teaching: 










Images or visual materials 


75% 


72% 


71% 


80% 


News or other media sources and archives 


64% 


55% 


74% 


81% 


“Portals” that provide links or URL’s relevant to particular disciplinary 
topics 


63% 


61% 


68% 


66% 


Online reference resources 


62% 


57% 


67% 


72% 


Digital film or video 


62% 


57% 


66% 


72% 


Maps 


53% 


52% 


53% 


51% 


Online or digitized documents 


50% 


52% 


60% 


43% 


Audio materials 


46% 


41% 


46% 


54% 


Curricular materials and websites that are created by other faculty 
and/or other institutions 


35% 


32% 


36% 


43% 


Digital readers or coursepacks 


30% 


29% 


59% 


20% 


Online class discussions 


28% 


29% 


30% 


27% 


Government documents 


27% 


23% 


38% 


35% 


Data archives 


27% 


24% 


27% 


31% 


Digital facsimiles of ancient or historical manuscripts 


23% 


24% 


19% 


20% 


Simulations or animations 


19% 


15% 


17% 


29% 


Personal online diaries 


9% 


7% 


8% 


12% 


How often do you use digital resources in your undergraduate 
teaching from each of the following sources? 










Search engines/directories 


81% 


77% 


88% 


87% 


My own personal collection of digital materials 


69% 


69% 


64% 


71% 


Online journals 


62% 


65% 


80% 


46% 


Public online image databases 


62% 


62% 


60% 


62% 


Media sites 


57% 


52% 


60% 


69% 


Library collections 


57% 


57% 


70% 


52% 


“Portals” that provide links or URL’s relevant to particular disciplinary 
topics 


55% 


53% 


59% 


58% 


Online exhibits 


37% 


36% 


43% 


37% 


Campus image databases from my own institution 


24% 


21% 


31% 


26% 


Commercial image databases 


9% 


5% 


10% 


15% 


How often have you heard about sources of digital resources from 
each of the following? 










Word of mouth from colleagues 


74% 


73% 


77% 


73% 


Word of mouth from students 


57% 


54% 


57% 


61% 


Professional societies or discussion lists 


55% 


57% 


69% 


46% 


Recommendation from a campus librarian 


51% 


49% 


53% 


57% 


A campus department devoted to instructional technology 


41% 


40% 


39% 


46% 


How often do you use digital resources in each of these ways? 










Presented during my lectures/class 


71% 


68% 


72% 


78% 


Assigned to students for review and/or study 


59% 


56% 


65% 


64% 


Assigned for student research projects or problem-based learning 
assignments 


56% 


50% 


68% 


65% 


Posted directly on my course website 


52% 


57% 


48% 


38% 


Linked from my course website 


49% 


53% 


48% 


39% 


Used in tests and quizzes 


25% 


20% 


24% 


37% 


Assigned to students to create their own digital portfolios and/or 
multimedia projects 


20% 


18% 


27% 


22% 


Presented in the context of an online discussion 


12% 


9% 


11% 


21% 
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All 


University 

of 

California 


Liberal 

arts 

colleges 


Community 

colleges 


Presented in my online lectures 


12% 


9% 


7% 


21% 


How much do you agree or disagree with the following statements 
about your reasons for using digital resources? i use digital 
resources in my teaching... 










because it improves my students’ learning. 


78% 


77% 


70% 


86% 


to integrate primary source material into the course. 


78% 


76% 


82% 


82% 


to provide students a context for a topic. 


75% 


72% 


77% 


81% 


to get students excited about a topic. 


73% 


69% 


75% 


84% 


because it allows me to do things in the classroom that 1 could never do 
otherwise. 


68% 


67% 


69% 


68% 


because it provides access to resources that we don’t have at our 
college. 


61% 


51% 


80% 


76% 


because it allows my students to be more creative. 


56% 


51% 


52% 


72% 


to teach critical thinking skills. 


56% 


47% 


61% 


75% 


to integrate my research interests into my course. 


55% 


53% 


60% 


59% 


because 1 like or feel very comfortable with the new technologies. 


53% 


49% 


55% 


63% 


because it is more convenient for my students and their schedules. 


51% 


50% 


49% 


55% 


to teach information literacy (i.e., evaluating the online materials 
themselves). 


43% 


36% 


54% 


56% 


because it saves me time. 


41% 


41% 


40% 


41% 


because my students expect or ask for more technology. 


38% 


35% 


38% 


43% 


because it allows me to stay up-to-date with my colleagues. 


37% 


31% 


36% 


55% 


to provide students with both good and bad examples of different kinds 
of scholarship. 


32% 


25% 


35% 


49% 


because it creates a sense of community for students enrolled in my 
course. 


32% 


29% 


29% 


40% 


because 1 enjoy having my teaching practices and course materials 
available to anyone in the world who would like to use them. 


25% 


22% 


25% 


31% 


because the administration encourages me to use digital resources 
more. 


23% 


18% 


28% 


32% 


to provide students a preview of the course before they register. 


22% 


21% 


13% 


27% 


because it may help me get promoted or get tenure. 


9% 


8% 


11% 


12% 


How much do you agree or disagree with the following statements 
about your reasons for not using digital resources? 1 don’t use 
digital resources in certain teaching situations, because... 










they cannot substitute for the teaching approaches 1 use. 


75% 


78% 


81% 


66% 


1 don’t have time to use digital resources. 


66% 


67% 


72% 


61% 


using them distracts from the core goals of my teaching. 


47% 


49% 


59% 


34% 


1 don’t want my students to copy or plagiarize material from the web. 


33% 


35% 


21% 


34% 


they are irrelevant to my field. 


30% 


30% 


38% 


28% 


students don’t have the information literacy skills to assess the 
credibility of digital resources. 


29% 


25% 


27% 


39% 


digital material can be presented outside its original context. 


25% 


24% 


19% 


31% 


How much do you agree or disagree with the following 
statements? 1 have difficulty using digital resources the way 1 
would like, because... 










1 don’t have reliable access to physical resources in my classroom(s). 


53% 


52% 


43% 


63% 


the digital resources are distributed in so many places that it is difficult 
for me to organize them for use in my teaching. 


45% 


43% 


56% 


46% 


there are too many resources out there for me to take advantage of — 1 
am overwhelmed. 


44% 


42% 


51% 


47% 


1 don’t have time to assess the credibility of the available resources. 


43% 


42% 


48% 


44% 


the content 1 need or want is just not available online. 


41% 


47% 


48% 


25% 


the academic quality of available materials is too poor to meet my 
needs. 


39% 


45% 


40% 


24% 


1 don’t have reliable access to scanners. 


39% 


40% 


34% 


39% 


1 don’t know how to locate the online materials 1 need. 


36% 


33% 


43% 


40% 
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All 


University 

of 

California 


Liberal 

arts 

colleges 


Community 

colleges 


1 don’t know how to save presentations to my computer so they can be 
run without a live connection. 


35% 


35% 


35% 


36% 


available software is unsuitable for integrating audio or video into my 
course. 


34% 


31% 


40% 


40% 


search engines provide irrelevant results for my needs. 


34% 


31% 


35% 


39% 


1 have difficulty understanding the issues surrounding copyright and 
digital collections. 


33% 


33% 


33% 


34% 


my students don't have a high-speed connection. 


32% 


28% 


13% 


54% 


course management software packages are inadequate for my needs. 


32% 


32% 


41% 


29% 


web sites 1 would use are unreliable, and 1 can’t count on them being 
there when 1 need them. 


32% 


30% 


39% 


33% 


available software is unsuitable for viewing and displaying digital 
images. 


31% 


29% 


39% 


31% 


my students don’t have reliable access to computers. 


30% 


24% 


10% 


54% 


web formats allow me to link to whole documents, but not to specific 
excerpts within a text. 


28% 


28% 


21% 


32% 


it is difficult to get server space or access to a server in order to 
store/host digital resources for teaching. 


27% 


23% 


21% 


38% 


1 don’t have reliable access to a high-speed connection. 


21% 


19% 


15% 


32% 


1 don’t have reliable access to a computer. 


13% 


10% 


11% 


21% 


How important is it for you to have support or assistance with each 
of the following activities for your teaching? 










obtaining or setting up technical infrastructure 


82% 


81% 


89% 


84% 


creating my own website 


82% 


81% 


87% 


81% 


digitizing existing resources 


80% 


79% 


92% 


78% 


learning how to use a learning management system 


79% 


78% 


85% 


80% 


importing resources into a course website or a database 


79% 


79% 


81% 


77% 


gathering, organizing, and maintaining digital materials 


78% 


77% 


86% 


77% 


integrating resources into a learning management system 


78% 


75% 


83% 


81% 


finding digital resources 


72% 


69% 


75% 


78% 


training students to find or evaluate digital resources 


71% 


67% 


77% 


79% 


interpreting copyright laws and/or securing copyright permission 


65% 


62% 


69% 


70% 


assessing the credibility of digital resources 


50% 


43% 


50% 


69% 


evaluating the appropriateness of resources for my teaching goals 


42% 


35% 


38% 


60% 


How satisfied or dissatisfied are you with the support you have 
received from the following sources? 










Campus librarians 


84% 


85% 


83% 


84% 


Friends or family 


83% 


81% 


76% 


90% 


Graduate students 


80% 


81% 


64% 


75% 


Campus educational technology or IT support staff 


73% 


74% 


76% 


68% 


Other faculty 


71% 


70% 


72% 


74% 


Undergraduate students 


70% 


73% 


65% 


69% 


My departmental or college staff 


66% 


68% 


66% 


63% 


Workshops 


60% 


55% 


59% 


67% 


Online help or guides 


47% 


44% 


48% 


53% 












Overall digital resource use (0-100) 


31 


29 


34 


32 


Overall technology use (0=no usage; 16=max usage) 


14.5 


14.8 


14.8 


13.6 












Principal Component Scores: 










What resources people use 










General purpose and reference materials 


0.00 


-0.03 


0.23 


-0.02 


Images and audiovisual materials 


0.02 


-0.15 


-0.28 


0.57 


Historical documents, maps, and primary sources 


-0.03 


0.06 


0.04 


-0.35 


Data, news/media, and governmental resources 


0.00 


-0.12 


0.23 


0.19 


Discussion and curricular materials 


0.00 


-0.01 


0.26 


-0.06 
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All 


University 

of 

California 


Liberal 

arts 

colleges 


Community 

colleges 


How people use digital resources 










Student assignments 


0.01 


-0.07 


0.20 


0.17 


Web posting 


0.00 


0.14 


-0.01 


-0.39 


Online courses 


0.00 


-0.13 


-0.14 


0.41 


In-class presentation 


0.01 


-0.05 


-0.11 


0.20 


Motivations for use or non-use 










Pedagogical reasons 


0.03 


-0.03 


0.04 


0.21 


Expectations and reputation 


-0.05 


-0.10 


-0.12 


0.12 


Inappropriateness 


0.02 


0.10 


0.33 


-0.33 


Concerns about students’ interpretation and information literacy 


0.01 


-0.06 


-0.09 


0.25 


Time, convenience, and access 


-0.02 


-0.05 


0.06 


0.01 


Teaching information literacy and critical thinking 


0.03 


-0.15 


0.24 


0.40 


Making information publicly available 


0.02 


0.06 


-0.12 


-0.04 


Using free and publicly available materials 


-0.06 


-0.04 


0.06 


-0.16 


Barriers to use 










Finding resources 


0.02 


-0.04 


0.24 


0.05 


High-end multimedia equipment and software 


0.00 


-0.02 


0.18 


0.01 


Personal equipment 


0.00 


-0.07 


-0.20 


0.28 


Resource availability or existence 


0.02 


0.11 


0.16 


-0.29 


Student equipment 


-0.01 


-0.11 


-0.59 


0.54 


Activities with which support is needed 










Technical activities 


0.00 


0.01 


0.27 


-0.15 


Intellectual and content-based activities 


0.01 


-0.12 


-0.03 


0.36 
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Table H.2: Faculty survey responses, broken out by discipline 





< 


Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


N 


831 


65 


102 


99 


90 


33 


93 


235 


18 


26 


13 


How often do you use the following types of digital 
resources in your undergraduate teaching: 
























Images or visual materials 


75% 


76% 


92% 


78% 


56% 


67% 


85% 


64% 


100% 


88% 


75% 


News or other media sources and archives 


64% 


63% 


57% 


63% 


77% 


81% 


70% 


59% 


72% 


63% 


67% 


“Portals” that provide links or URL’s relevant to 
particular disciplinary topics 


63% 


69% 


75% 


61% 


52% 


76% 


60% 


60% 


56% 


54% 


75% 


Online reference resources 


62% 


48% 


61% 


62% 


42% 


72% 


77% 


71% 


61% 


54% 


50% 


Digital film or video 


62% 


65% 


77% 


61% 


44% 


69% 


63% 


59% 


44% 


76% 


82% 


Maps 


53% 


69% 


44% 


81% 


50% 


23% 


69% 


37% 


100% 


65% 


8% 


Online or digitized documents 


50% 


48% 


46% 


68% 


48% 


48% 


52% 


46% 


50% 


50% 


42% 


Audio materials 


46% 


26% 


38% 


44% 


30% 


55% 


69% 


52% 


11% 


65% 


50% 


Curricular materials and websites that are created by 
other faculty and/or other institutions 


35% 


34% 


46% 


28% 


21% 


47% 


49% 


31% 


44% 


29% 


50% 


Digital readers or coursepacks 


30% 


49% 


32% 


37% 


35% 


32% 


27% 


18% 


28% 


28% 


36% 


Online class discussions 


28% 


35% 


33% 


31% 


28% 


57% 


17% 


25% 


22% 


25% 


42% 


Government documents 


27% 


26% 


11% 


51% 


58% 


24% 


9% 


15% 


44% 


40% 


25% 


Data archives 


27% 


27% 


16% 


30% 


58% 


21% 


15% 


12% 


83% 


50% 


25% 


Digital facsimiles of ancient or historical manuscripts 


23% 


23% 


17% 


49% 


12% 


10% 


16% 


24% 


11% 


25% 


0% 


Simulations or animations 


19% 


21% 


44% 


14% 


10% 


17% 


23% 


9% 


44% 


13% 


8% 


Personal online diaries 


9% 


5% 


15% 


9% 


6% 


25% 


7% 


8% 


0% 


4% 


25% 


How often do you use digital resources in your 
undergraduate teaching from each of the following 
sources? 
























Search engines/directories 


81% 


76% 


77% 


86% 


68% 


91% 


83% 


83% 


89% 


77% 


83% 


My own personal collection of digital materials 


69% 


81% 


81% 


74% 


67% 


58% 


69% 


58% 


83% 


84% 


83% 


Online journals 


62% 


77% 


42% 


75% 


81% 


71% 


33% 


61% 


61% 


63% 


67% 


Public online image databases 


62% 


61% 


68% 


71% 


48% 


53% 


65% 


57% 


82% 


64% 


67% 


Media sites 


57% 


57% 


53% 


55% 


80% 


73% 


50% 


52% 


56% 


50% 


58% 


Library collections 


57% 


58% 


53% 


72% 


56% 


77% 


39% 


59% 


44% 


60% 


67% 
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< 


Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


“Portals” that provide links or URL’s relevant to particular 
disciplinary topics 


55% 


65% 


62% 


57% 


48% 


67% 


50% 


50% 


44% 


56% 


58% 


Online exhibits 


37% 


37% 


67% 


65% 


11% 


40% 


35% 


27% 


17% 


40% 


17% 


Campus image databases from my own institution 


24% 


20% 


47% 


25% 


12% 


23% 


22% 


18% 


28% 


16% 


25% 


Commercial image databases 


9% 


7% 


24% 


8% 


1% 


10% 


7% 


6% 


0% 


4% 


17% 


How often have you heard about sources of digital 
resources from each of the following? 
























Word of mouth from colleagues 


74% 


78% 


70% 


78% 


67% 


91% 


74% 


73% 


72% 


65% 


75% 


Word of mouth from students 


57% 


51% 


71% 


56% 


41% 


66% 


53% 


55% 


67% 


65% 


75% 


Professional societies or discussion lists 


55% 


52% 


47% 


77% 


55% 


50% 


54% 


52% 


44% 


62% 


58% 


Recommendation from a campus librarian 


51% 


48% 


40% 


61% 


42% 


72% 


38% 


61% 


22% 


62% 


58% 


A campus department devoted to instructional 
technology 


41% 


33% 


43% 


38% 


24% 


44% 


56% 


43% 


50% 


46% 


27% 


How often do you use digital resources in each of 
these ways? 
























Presented during my lectures/class 


71% 


84% 


81% 


84% 


67% 


61% 


75% 


57% 


82% 


84% 


75% 


Assigned to students for review and/or study 


59% 


56% 


71% 


52% 


58% 


53% 


70% 


55% 


39% 


48% 


67% 


Assigned for student research projects or problem-based 
learning assignments 


56% 


52% 


57% 


55% 


52% 


63% 


62% 


55% 


78% 


48% 


75% 


Posted directly on my course website 


52% 


63% 


60% 


52% 


60% 


48% 


57% 


38% 


56% 


58% 


50% 


Linked from my course website 


49% 


65% 


57% 


49% 


59% 


41% 


55% 


36% 


50% 


50% 


55% 


Used in tests and quizzes 


25% 


22% 


37% 


27% 


19% 


29% 


36% 


16% 


28% 


12% 


42% 


Assigned to students to create their own digital portfolios 
and/or multimedia projects 


20% 


15% 


42% 


20% 


5% 


27% 


21% 


15% 


28% 


15% 


17% 


Presented in the context of an online discussion 


12% 


15% 


18% 


13% 


7% 


30% 


8% 


9% 


13% 


8% 


25% 


Presented in my online lectures 


12% 


15% 


20% 


12% 


8% 


13% 


7% 


6% 


35% 


8% 


33% 


How much do you agree or disagree with the 
following statements about your reasons for using 
digital resources? 1 use digital resources in my 
teaching... 
























because it improves my students’ learning. 


78% 


85% 


82% 


81% 


70% 


80% 


90% 


71% 


88% 


88% 


83% 


to integrate primary source material into the course. 


78% 


80% 


87% 


89% 


73% 


73% 


85% 


71% 


78% 


72% 


82% 
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< 


Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


to provide students a context for a topic. 


75% 


77% 


77% 


84% 


65% 


69% 


82% 


73% 


83% 


80% 


45% 


to get students excited about a topic. 


73% 


79% 


71% 


84% 


64% 


73% 


81% 


68% 


78% 


76% 


55% 


because it allows me to do things in the classroom that 1 
could never do otherwise. 


68% 


66% 


73% 


79% 


63% 


66% 


79% 


56% 


78% 


72% 


67% 


because it provides access to resources that we don’t 
have at our college. 


61% 


54% 


76% 


63% 


49% 


61% 


70% 


55% 


72% 


46% 


67% 


because it allows my students to be more creative. 


56% 


51% 


74% 


53% 


37% 


53% 


67% 


53% 


61% 


58% 


75% 


to teach critical thinking skills. 


56% 


62% 


58% 


63% 


45% 


83% 


48% 


55% 


61% 


52% 


50% 


to integrate my research interests into my course. 


55% 


68% 


60% 


47% 


58% 


30% 


54% 


51% 


72% 


76% 


82% 


because 1 like or feel very comfortable with the new 
technologies. 


53% 


55% 


72% 


61% 


46% 


60% 


63% 


43% 


78% 


28% 


58% 


because it is more convenient for my students and their 
schedules. 


51% 


49% 


64% 


40% 


55% 


72% 


52% 


44% 


72% 


40% 


42% 


to teach information literacy (i.e., evaluating the online 
materials themselves). 


43% 


43% 


47% 


43% 


24% 


80% 


33% 


48% 


67% 


28% 


40% 


because it saves me time. 


41% 


45% 


53% 


33% 


44% 


59% 


39% 


35% 


39% 


35% 


42% 


because my students expect or ask for more technology. 


38% 


36% 


48% 


41% 


28% 


50% 


34% 


32% 


44% 


33% 


50% 


because it allows me to stay up-to-date with my 
colleagues. 


37% 


30% 


52% 


29% 


21% 


53% 


45% 


37% 


50% 


29% 


45% 


to provide students with both good and bad examples of 
different kinds of scholarship. 


32% 


36% 


43% 


29% 


23% 


57% 


20% 


33% 


39% 


17% 


60% 


because it creates a sense of community for students 
enrolled in my course. 


32% 


27% 


39% 


22% 


19% 


59% 


38% 


33% 


28% 


20% 


58% 


because 1 enjoy having my teaching practices and 
course materials available to anyone in the world who 
would like to use them. 


25% 


25% 


36% 


23% 


23% 


30% 


34% 


16% 


39% 


33% 


18% 


because the administration encourages me to use digital 
resources more. 


23% 


21% 


30% 


20% 


8% 


30% 


26% 


24% 


17% 


24% 


25% 


to provide students a preview of the course before they 
register. 


22% 


34% 


30% 


18% 


14% 


20% 


23% 


19% 


22% 


0% 


40% 


because it may help me get promoted or get tenure. 


9% 


13% 


16% 


7% 


3% 


20% 


13% 


6% 


0% 


4% 


8% 
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< 


Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


How much do you agree or disagree with the 
following statements about your reasons for not 
using digital resources? 1 don’t use digital 
resources in certain teaching situations, because... 
























they cannot substitute for the teaching approaches 1 use. 


75% 


64% 


61% 


79% 


73% 


90% 


83% 


84% 


39% 


76% 


58% 


1 don’t have time to use digital resources. 


66% 


57% 


57% 


68% 


70% 


58% 


73% 


74% 


28% 


64% 


50% 


using them distracts from the core goals of my teaching. 


47% 


51% 


34% 


51% 


53% 


55% 


45% 


51% 


11% 


39% 


50% 


1 don’t want my students to copy or plagiarize material 
from the web. 


33% 


41% 


25% 


27% 


26% 


42% 


34% 


37% 


28% 


33% 


33% 


they are irrelevant to my field. 


30% 


18% 


19% 


29% 


24% 


43% 


41% 


40% 


11% 


28% 


25% 


students don’t have the information literacy skills to 
assess the credibility of digital resources. 


29% 


31% 


26% 


31% 


21% 


30% 


27% 


32% 


11% 


39% 


18% 


digital material can be presented outside its original 
context. 


25% 


25% 


15% 


21% 


22% 


26% 


31% 


28% 


11% 


32% 


18% 


How much do you agree or disagree with the 
following statements? 1 have difficulty using digital 
resources the way 1 would like, because... 
























1 don’t have reliable access to physical resources in my 
classroom(s). 


53% 


53% 


49% 


51% 


48% 


62% 


61% 


55% 


39% 


63% 


42% 


the digital resources are distributed in so many places 
that it is difficult for me to organize them for use in my 
teaching. 


45% 


38% 


39% 


44% 


36% 


48% 


55% 


51% 


28% 


32% 


58% 


there are too many resources out there for me to take 
advantage of — 1 am overwhelmed. 


44% 


41% 


25% 


41% 


38% 


62% 


43% 


54% 


44% 


52% 


25% 


1 don’t have time to assess the credibility of the available 
resources. 


43% 


46% 


35% 


41% 


39% 


45% 


59% 


42% 


50% 


48% 


25% 


the content 1 need or want is just not available online. 


41% 


44% 


41% 


38% 


34% 


36% 


45% 


43% 


22% 


60% 


50% 


the academic quality of available materials is too poor to 
meet my needs. 


39% 


46% 


44% 


34% 


27% 


36% 


36% 


42% 


33% 


42% 


42% 


1 don’t have reliable access to scanners. 


39% 


30% 


31% 


46% 


35% 


33% 


52% 


39% 


39% 


38% 


42% 


1 don’t know how to locate the online materials 1 need. 


36% 


39% 


29% 


35% 


32% 


34% 


40% 


41% 


12% 


36% 


50% 


1 don’t know how to save presentations to my computer 
so they can be run without a live connection. 


35% 


32% 


29% 


33% 


32% 


36% 


38% 


42% 


22% 


38% 


33% 
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< 


Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


available software is unsuitable for integrating audio or 
video into my course. 


34% 


33% 


31% 


37% 


35% 


41% 


49% 


29% 


28% 


38% 


25% 


search engines provide irrelevant results for my needs. 


34% 


33% 


29% 


33% 


28% 


32% 


43% 


39% 


22% 


38% 


17% 


1 have difficulty understanding the issues surrounding 
copyright and digital collections. 


33% 


31% 


38% 


25% 


19% 


24% 


54% 


30% 


33% 


42% 


42% 


my students don't have a high-speed connection. 


32% 


37% 


29% 


30% 


24% 


28% 


37% 


35% 


33% 


48% 


17% 


course management software packages are inadequate 
for my needs. 


32% 


36% 


37% 


30% 


26% 


16% 


41% 


29% 


35% 


48% 


17% 


web sites 1 would use are unreliable, and 1 can’t count on 
them being there when 1 need them. 


32% 


25% 


30% 


36% 


29% 


28% 


44% 


31% 


28% 


33% 


25% 


available software is unsuitable for viewing and 
displaying digital images. 


31% 


24% 


41% 


32% 


29% 


34% 


38% 


28% 


17% 


26% 


25% 


my students don’t have reliable access to computers. 


30% 


33% 


26% 


31% 


15% 


34% 


38% 


31% 


22% 


61% 


8% 


web formats allow me to link to whole documents, but 
not to specific excerpts within a text. 


28% 


24% 


24% 


23% 


25% 


32% 


33% 


28% 


33% 


55% 


17% 


it is difficult to get server space or access to a server in 
order to store/host digital resources for teaching. 


27% 


29% 


36% 


21% 


19% 


12% 


39% 


22% 


28% 


35% 


25% 


1 don’t have reliable access to a high-speed connection. 


21% 


22% 


20% 


23% 


11% 


21% 


26% 


24% 


11% 


22% 


17% 


1 don’t have reliable access to a computer. 


13% 


10% 


16% 


12% 


7% 


14% 


20% 


12% 


0% 


17% 


8% 


How important is it for you to have support or 
assistance with each of the following activities for 
your teaching? 
























obtaining or setting up technical infrastructure 


82% 


84% 


84% 


87% 


82% 


70% 


90% 


76% 


72% 


91% 


92% 


creating my own website 


82% 


74% 


81% 


90% 


74% 


72% 


86% 


83% 


72% 


87% 


92% 


digitizing existing resources 


80% 


82% 


82% 


89% 


69% 


62% 


90% 


78% 


67% 


83% 


92% 


learning how to use a learning management system 


79% 


80% 


85% 


82% 


67% 


62% 


93% 


78% 


67% 


83% 


75% 


importing resources into a course website or a database 


79% 


75% 


76% 


87% 


74% 


64% 


85% 


79% 


72% 


87% 


92% 


gathering, organizing, and maintaining digital materials 


78% 


75% 


85% 


78% 


72% 


59% 


85% 


77% 


56% 


87% 


92% 


integrating resources into a learning management 
system 


78% 


78% 


84% 


81% 


64% 


64% 


91% 


76% 


61% 


87% 


75% 


finding digital resources 


72% 


66% 


80% 


70% 


63% 


72% 


81% 


73% 


61% 


70% 


83% 


training students to find or evaluate digital resources 


71% 


70% 


78% 


75% 


59% 


52% 


73% 


71% 


61% 


87% 


50% 
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Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


interpreting copyright laws and/or securing copyright 
permission 


65% 


62% 


66% 


67% 


54% 


41% 


80% 


62% 


56% 


87% 


67% 


assessing the credibility of digital resources 


50% 


36% 


59% 


44% 


36% 


41% 


64% 


53% 


56% 


65% 


58% 


evaluating the appropriateness of resources for my 
teaching goals 


42% 


37% 


41% 


40% 


26% 


29% 


58% 


45% 


50% 


57% 


50% 


How satisfied or dissatisfied are you with the 
support you have received from the following 
sources? 
























Campus librarians 


84% 


87% 


82% 


82% 


75% 


95% 


78% 


89% 


80% 


91% 


86% 


Friends or family 


83% 


83% 


79% 


85% 


88% 


93% 


88% 


80% 


88% 


85% 


100% 


Graduate students 


80% 


92% 


80% 


79% 


83% 


33% 


72% 


77% 


91% 


71% 


100% 


Campus educational technology or IT support staff 


73% 


77% 


58% 


72% 


74% 


88% 


74% 


79% 


85% 


67% 


64% 


Other faculty 


71% 


81% 


58% 


74% 


72% 


80% 


70% 


75% 


100% 


77% 


86% 


Undergraduate students 


70% 


78% 


64% 


84% 


67% 


80% 


59% 


72% 


71% 


64% 


100% 


My departmental or college staff 


66% 


70% 


60% 


64% 


77% 


80% 


62% 


65% 


69% 


67% 


60% 


Workshops 


60% 


57% 


51% 


52% 


46% 


78% 


70% 


66% 


36% 


43% 


75% 


Online help or guides 


47% 


41% 


53% 


41% 


58% 


65% 


48% 


45% 


46% 


19% 


40% 


























Overall digital resource use (0-100) 


31 


33 


33 


36 


28 


32 


32 


27 


42 


31 


31 


Overall technology use (0=no usage; 16=max usage) 


14.5 


14.8 


14.0 


15.1 


14.5 


14.2 


14.3 


14.5 


14.4 


14.7 


15.0 


























Principal Component Scores: 
























What resources people use 
























General purpose and reference materials 


0.00 


-0.12 


0.05 


-0.05 


-0.20 


0.44 


-0.09 


0.15 


-0.35 


-0.34 


0.16 


Images and audiovisual materials 


0.02 


-0.18 


0.89 


-0.18 


-0.74 


0.00 


0.53 


-0.13 


0.20 


-0.02 


0.51 


Historical documents, maps, and primary sources 


-0.03 


0.35 


-0.01 


0.88 


-0.16 


-0.75 


-0.31 


-0.32 


0.51 


0.28 


-0.77 


Data, news/media, and governmental resources 


0.00 


0.00 


-0.67 


0.27 


0.99 


0.03 


-0.25 


-0.28 


0.97 


0.45 


0.00 


Discussion and curricular materials 


0.00 


0.19 


0.11 


0.02 


0.02 


0.48 


-0.11 


-0.08 


-0.38 


-0.26 


0.48 


How people use digital resources 
























Student assignments 


0.01 


-0.23 


0.52 


-0.18 


-0.31 


0.32 


0.13 


0.03 


-0.03 


-0.37 


0.14 


Web posting 


0.00 


0.40 


0.06 


0.05 


0.29 


-0.21 


0.12 


-0.28 


0.01 


0.10 


-0.22 
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Anthropology & 
Archaeology 


Art & Architecture 


History 


Political science 


Writing 


Foreign language 


Literature & English 
language 


Geography 


Ethnic, gender, and 
cultural studies 


Media studies & 
communications 


Online courses 


0.00 


-0.05 


0.12 


-0.06 


-0.08 


0.44 


-0.12 


-0.07 


0.28 


-0.08 


0.55 


In-class presentation 


0.01 


0.24 


0.31 


0.47 


-0.17 


-0.51 


0.01 


-0.31 


0.62 


0.17 


0.18 


Motivations for use or non-use 
























Pedagogical reasons 


0.03 


0.05 


0.20 


0.31 


-0.27 


-0.10 


0.39 


-0.18 


0.21 


0.27 


-0.18 


Expectations and reputation 


-0.05 


-0.07 


0.09 


-0.22 


-0.40 


0.54 


0.06 


-0.01 


-0.30 


-0.19 


0.16 


Inappropriateness 


0.02 


-0.25 


-0.32 


0.08 


0.09 


0.31 


0.19 


0.26 


-0.98 


-0.03 


-0.46 


Concerns about students’ interpretation and information 
literacy 


0.01 


0.01 


-0.11 


-0.03 


-0.21 


0.11 


0.03 


0.07 


-0.02 


0.36 


0.13 


Time, convenience, and access 


-0.02 


-0.14 


0.40 


-0.25 


0.10 


0.22 


0.12 


-0.22 


0.26 


-0.35 


0.09 


Teaching information literacy and critical thinking 


0.03 


0.04 


0.10 


0.07 


-0.17 


0.61 


-0.40 


0.21 


0.26 


-0.14 


0.18 


Making information publicly available 


0.02 


0.39 


0.08 


-0.10 


-0.02 


-0.14 


0.06 


-0.11 


0.12 


0.28 


0.35 


Using free and publicly available materials 


-0.06 


0.16 


0.03 


0.07 


-0.07 


-0.44 


-0.10 


-0.16 


0.11 


-0.19 


-0.35 


Barriers to use 
























Finding resources 


0.02 


-0.13 


-0.22 


-0.10 


-0.03 


0.10 


0.22 


0.18 


-0.15 


-0.03 


-0.14 


High-end multimedia equipment and software 


0.00 


-0.09 


0.10 


0.09 


0.09 


-0.14 


0.22 


-0.14 


-0.06 


0.07 


-0.12 


Personal equipment 


0.00 


-0.05 


-0.07 


-0.04 


-0.20 


-0.07 


0.18 


0.04 


-0.17 


0.27 


-0.11 


Resource availability or existence 


0.02 


0.25 


0.07 


-0.03 


-0.22 


-0.25 


0.08 


0.03 


-0.15 


0.17 


0.08 


Student equipment 


-0.01 


0.11 


-0.04 


-0.10 


-0.32 


0.12 


0.10 


0.03 


0.03 


0.39 


-0.24 


Activities with which support is needed 
























Technical activities 


0.00 


-0.01 


0.07 


0.23 


-0.13 


-0.20 


0.22 


-0.07 


-0.53 


0.21 


-0.39 


Intellectual and content-based activities 


0.01 


-0.18 


0.18 


-0.04 


-0.41 


-0.30 


0.34 


0.08 


-0.03 


0.22 


-0.04 
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Table H.3: Faculty survey responses, broken out by age group 





All 


00-35 


36-44 


45-53 


54-61 


62+ 


N 


831 


76 


169 


181 


184 


120 


How often do you use the following types of digital resources 
in your undergraduate teaching: 














Images or visual materials 


75% 


76% 


78% 


77% 


76% 


61% 


News or other media sources and archives 


64% 


70% 


64% 


65% 


61% 


56% 


“Portals” that provide links or URL’s relevant to particular 
disciplinary topics 


63% 


64% 


66% 


65% 


66% 


46% 


Online reference resources 


62% 


70% 


59% 


67% 


62% 


45% 


Digital film or video 


62% 


67% 


60% 


62% 


60% 


56% 


Maps 


53% 


56% 


55% 


52% 


55% 


44% 


Online or digitized documents 


50% 


59% 


57% 


53% 


46% 


41% 


Audio materials 


46% 


47% 


52% 


40% 


45% 


44% 


Curricular materials and websites that are created by other faculty 
and/or other institutions 


35% 


38% 


39% 


37% 


32% 


23% 


Digital readers or coursepacks 


30% 


38% 


33% 


31% 


31% 


23% 


Online class discussions 


28% 


29% 


31% 


30% 


30% 


22% 


Government documents 


27% 


34% 


28% 


26% 


25% 


25% 


Data archives 


27% 


37% 


26% 


22% 


26% 


22% 


Digital facsimiles of ancient or historical manuscripts 


23% 


24% 


22% 


21% 


28% 


18% 


Simulations or animations 


19% 


26% 


18% 


14% 


22% 


13% 


Personal online diaries 


9% 


13% 


8% 


9% 


7% 


5% 


How often do you use digital resources in your undergraduate 
teaching from each of the following sources? 














Search engines/directories 


81% 


82% 


81% 


81% 


85% 


67% 


My own personal collection of digital materials 


69% 


72% 


67% 


73% 


72% 


64% 


Online journals 


62% 


79% 


69% 


66% 


58% 


47% 


Public online image databases 


62% 


65% 


67% 


60% 


66% 


49% 


Media sites 


57% 


67% 


61% 


53% 


57% 


52% 


Library collections 


57% 


67% 


62% 


59% 


57% 


44% 


“Portals” that provide links or URL’s relevant to particular 
disciplinary topics 


55% 


53% 


58% 


55% 


60% 


44% 


Online exhibits 


37% 


35% 


44% 


39% 


41% 


23% 


Campus image databases from my own institution 


24% 


21% 


24% 


20% 


29% 


20% 


Commercial image databases 


9% 


11% 


6% 


7% 


11% 


7% 


How often have you heard about sources of digital resources 
from each of the following? 














Word of mouth from colleagues 


74% 


81% 


75% 


74% 


74% 


67% 


Word of mouth from students 


57% 


46% 


52% 


62% 


66% 


50% 


Professional societies or discussion lists 


55% 


49% 


55% 


61% 


56% 


53% 


Recommendation from a campus librarian 


51% 


43% 


54% 


47% 


58% 


51% 


A campus department devoted to instructional technology 


41% 


36% 


44% 


38% 


47% 


39% 


How often do you use digital resources in each of these ways? 














Presented during my lectures/class 


71% 


84% 


76% 


71% 


73% 


57% 


Assigned to students for review and/or study 


59% 


61% 


60% 


62% 


60% 


51% 


Assigned for student research projects or problem-based learning 
assignments 


56% 


66% 


53% 


58% 


60% 


47% 


Posted directly on my course website 


52% 


54% 


55% 


50% 


56% 


43% 


Linked from my course website 


49% 


54% 


54% 


49% 


52% 


35% 


Used in tests and quizzes 


25% 


31% 


25% 


23% 


26% 


21% 


Assigned to students to create their own digital portfolios and/or 
multimedia projects 


20% 


23% 


17% 


21% 


19% 


16% 


Presented in the context of an online discussion 


12% 


17% 


12% 


10% 


13% 


12% 


Presented in my online lectures 


12% 


18% 


6% 


10% 


15% 


10% 


How much do you agree or disagree with the following 
statements about your reasons for using digital resources? 1 
use digital resources in my teaching... 
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00-35 


36-44 


45-53 


54-61 


62+ 


because it improves my students’ learning. 


78% 


85% 


78% 


82% 


75% 


75% 


to integrate primary source material into the course. 


78% 


79% 


81% 


80% 


79% 


68% 


to provide students a context tor a topic. 


75% 


86% 


79% 


79% 


71% 


60% 


to get students excited about a topic. 


73% 


84% 


77% 


77% 


73% 


56% 


because it allows me to do things in the classroom that 1 could 
never do otherwise. 


68% 


70% 


68% 


73% 


66% 


57% 


because it provides access to resources that we don’t have at our 
college. 


61% 


58% 


59% 


63% 


66% 


51% 


because it allows my students to be more creative. 


56% 


63% 


59% 


53% 


57% 


45% 


to teach critical thinking skills. 


56% 


70% 


56% 


59% 


55% 


41% 


to integrate my research interests into my course. 


55% 


68% 


57% 


54% 


55% 


46% 


because 1 like or feel very comfortable with the new technologies. 


53% 


70% 


53% 


54% 


53% 


43% 


because it is more convenient for my students and their schedules. 


51% 


60% 


51% 


51% 


53% 


44% 


to teach information literacy (i.e., evaluating the online materials 
themselves). 


43% 


51% 


52% 


41% 


41% 


29% 


because it saves me time. 


41% 


51% 


41% 


41% 


40% 


41% 


because my students expect or ask for more technology. 


38% 


37% 


40% 


30% 


45% 


28% 


because it allows me to stay up-to-date with my colleagues. 


37% 


41% 


37% 


34% 


40% 


33% 


to provide students with both good and bad examples of different 
kinds of scholarship. 


32% 


47% 


38% 


25% 


32% 


25% 


because it creates a sense of community for students enrolled in 
my course. 


32% 


34% 


31% 


31% 


31% 


27% 


because 1 enjoy having my teaching practices and course materials 
available to anyone in the world who would like to use them. 


25% 


30% 


24% 


22% 


21% 


27% 


because the administration encourages me to use digital resources 
more. 


23% 


22% 


23% 


22% 


29% 


16% 


to provide students a preview of the course before they register. 


22% 


30% 


19% 


20% 


24% 


17% 


because it may help me get promoted or get tenure. 


9% 


11% 


14% 


9% 


6% 


4% 


How much do you agree or disagree with the following 
statements about your reasons for not using digital 
resources? 1 don’t use digital resources in certain teaching 
situations, because... 














they cannot substitute for the teaching approaches 1 use. 


75% 


66% 


80% 


76% 


75% 


75% 


1 don’t have time to use digital resources. 


66% 


59% 


74% 


69% 


66% 


58% 


using them distracts from the core goals of my teaching. 


47% 


45% 


45% 


50% 


49% 


45% 


1 don’t want my students to copy or plagiarize material from the 
web. 


33% 


36% 


34% 


30% 


34% 


30% 


they are irrelevant to my field. 


30% 


31% 


25% 


33% 


36% 


28% 


students don’t have the information literacy skills to assess the 
credibility of digital resources. 


29% 


24% 


31% 


28% 


33% 


25% 


digital material can be presented outside its original context. 


25% 


24% 


24% 


19% 


29% 


25% 


How much do you agree or disagree with the following 
statements? 1 have difficulty using digital resources the way 1 
would like, because... 














1 don’t have reliable access to physical resources in my 
classroom(s). 


53% 


48% 


53% 


58% 


58% 


42% 


the digital resources are distributed in so many places that it is 
difficult for me to organize them for use in my teaching. 


45% 


50% 


46% 


43% 


43% 


47% 


there are too many resources out there for me to take advantage 
of — 1 am overwhelmed. 


44% 


42% 


45% 


42% 


45% 


47% 


1 don’t have time to assess the credibility of the available resources. 


43% 


37% 


44% 


45% 


43% 


44% 


the content 1 need or want is just not available online. 


41% 


39% 


43% 


43% 


38% 


39% 


the academic quality of available materials is too poor to meet my 
needs. 


39% 


38% 


36% 


45% 


39% 


34% 


1 don’t have reliable access to scanners. 


39% 


38% 


43% 


40% 


41% 


29% 


1 don’t know how to locate the online materials 1 need. 


36% 


32% 


33% 


33% 


39% 


44% 


1 don’t know how to save presentations to my computer so they can 
be run without a live connection. 


35% 


22% 


32% 


39% 


35% 


40% 


available software is unsuitable for integrating audio or video into 
my course. 


34% 


27% 


40% 


30% 


38% 


25% 
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search engines provide irrelevant results tor my needs. 


34% 


32% 


32% 


31% 


36% 


37% 


1 have difficulty understanding the issues surrounding copyright and 
digital collections. 


33% 


30% 


34% 


30% 


32% 


36% 


my students don't have a high-speed connection. 


32% 


33% 


35% 


32% 


35% 


22% 


course management software packages are inadequate for my 
needs. 


32% 


28% 


32% 


32% 


32% 


35% 


web sites 1 would use are unreliable, and 1 can’t count on them 
being there when 1 need them. 


32% 


31% 


31% 


31% 


36% 


27% 


available software is unsuitable for viewing and displaying digital 
images. 


31% 


24% 


32% 


29% 


32% 


26% 


my students don’t have reliable access to computers. 


30% 


32% 


29% 


33% 


28% 


23% 


web formats allow me to link to whole documents, but not to specific 
excerpts within a text. 


28% 


19% 


26% 


28% 


33% 


29% 


it is difficult to get server space or access to a server in order to 
store/host digital resources for teaching. 


27% 


28% 


36% 


24% 


23% 


16% 


1 don’t have reliable access to a high-speed connection. 


21% 


16% 


20% 


23% 


23% 


21% 


1 don’t have reliable access to a computer. 


13% 


7% 


10% 


15% 


14% 


14% 


How important is it for you to have support or assistance with 
each of the following activities for your teaching? 














obtaining or setting up technical infrastructure 


82% 


84% 


88% 


82% 


85% 


67% 


creating my own website 


82% 


85% 


86% 


84% 


77% 


78% 


digitizing existing resources 


80% 


84% 


81% 


83% 


80% 


72% 


learning how to use a learning management system 


79% 


84% 


82% 


80% 


79% 


69% 


importing resources into a course website or a database 


79% 


80% 


81% 


84% 


78% 


69% 


gathering, organizing, and maintaining digital materials 


78% 


84% 


79% 


80% 


76% 


74% 


integrating resources into a learning management system 


78% 


84% 


83% 


79% 


77% 


63% 


finding digital resources 


72% 


65% 


74% 


70% 


75% 


71% 


training students to find or evaluate digital resources 


71% 


80% 


74% 


68% 


74% 


60% 


interpreting copyright laws and/or securing copyright permission 


65% 


75% 


66% 


63% 


68% 


52% 


assessing the credibility of digital resources 


50% 


49% 


52% 


49% 


50% 


50% 


evaluating the appropriateness of resources for my teaching goals 


42% 


45% 


40% 


38% 


39% 


47% 


How satisfied or dissatisfied are you with the support you have 
received from the following sources? 














Campus librarians 


84% 


77% 


82% 


83% 


84% 


94% 


Friends or family 


83% 


100% 


84% 


85% 


79% 


71% 


Graduate students 


80% 


75% 


79% 


77% 


78% 


91% 


Campus educational technology or IT support staff 


73% 


70% 


73% 


75% 


68% 


79% 


Other faculty 


71% 


79% 


64% 


73% 


68% 


77% 


Undergraduate students 


70% 


76% 


76% 


69% 


59% 


76% 


My departmental or college staff 


66% 


66% 


69% 


69% 


58% 


71% 


Workshops 


60% 


66% 


60% 


62% 


56% 


55% 


Online help or guides 


47% 


65% 


50% 


50% 


35% 


41% 
















Overall digital resource use (0-100) 


31 


34 


33 


31 


32 


25 


Overall technology use (0=no usage; 16=max usage) 


14.5 


14.8 


14.9 


14.7 


14.5 


13.6 
















Principal Component Scores: 














What resources people use 














General purpose and reference materials 


0.00 


0.09 


0.12 


0.04 


0.03 


-0.35 


Images and audiovisual materials 


0.02 


0.04 


-0.03 


-0.03 


0.10 


-0.15 


Historical documents, maps, and primary sources 


-0.03 


-0.01 


0.09 


-0.01 


0.08 


-0.25 


Data, news/media, and governmental resources 


0.00 


0.32 


0.04 


-0.13 


-0.08 


-0.08 


Discussion and curricular materials 


0.00 


0.18 


0.06 


0.00 


-0.01 


-0.15 


How people use digital resources 














Student assignments 


0.01 


0.09 


-0.02 


0.05 


0.08 


-0.13 


Web posting 


0.00 


0.06 


0.11 


0.05 


0.00 


-0.23 


Online courses 


0.00 


0.14 


-0.11 


-0.09 


0.01 


0.11 
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All 


00-35 


36-44 


45-53 


54-61 


62+ 


In-class presentation 


0.01 


0.21 


0.09 


-0.04 


0.06 


-0.22 


Motivations for use or non-use 














Pedagogical reasons 


0.03 


0.21 


0.06 


0.14 


-0.01 


-0.29 


Expectations and reputation 


-0.05 


-0.01 


0.09 


-0.12 


-0.03 


-0.26 


Inappropriateness 


0.02 


-0.10 


0.13 


0.19 


-0.03 


-0.21 


Concerns about students’ interpretation and information literacy 


0.01 


0.01 


-0.02 


-0.08 


0.11 


-0.05 


Time, convenience, and access 


-0.02 


0.07 


-0.03 


-0.07 


-0.02 


0.06 


Teaching information literacy and critical thinking 


0.03 


0.25 


0.13 


-0.09 


0.04 


-0.10 


Making information publicly available 


0.02 


0.26 


-0.04 


-0.02 


0.06 


-0.06 


Using free and publicly available materials 


-0.06 


-0.03 


0.01 


0.03 


-0.05 


-0.30 


Barriers to use 














Finding resources 


0.02 


0.00 


-0.01 


0.03 


0.00 


0.13 


High-end multimedia equipment and software 


0.00 


-0.11 


0.11 


-0.04 


0.06 


-0.19 


Personal equipment 


0.00 


0.04 


-0.01 


0.06 


-0.03 


-0.06 


Resource availability or existence 


0.02 


0.08 


0.01 


0.09 


0.00 


-0.08 


Student equipment 


-0.01 


0.01 


0.04 


0.01 


0.06 


-0.31 


Activities with which support is needed 














Technical activities 


0.00 


0.07 


0.15 


0.09 


-0.02 


-0.36 


Intellectual and content-based activities 


0.01 


-0.01 


-0.03 


-0.06 


0.03 


0.10 
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Table H.4: Faculty survey responses, broken out by level of use of digital resources 





All 


non- 

light 

users 


light- 

medium 

users 


medium- 

heavy 

users 


heavy 

users 


N 


831 


113 


306 


259 


153 


How often do you use the following types of digital resources in 
your undergraduate teaching: 












Images or visual materials 


75% 


21% 


70% 


88% 


97% 


News or other media sources and archives 


64% 


20% 


56% 


75% 


90% 


“Portals” that provide links or URL’s relevant to particular disciplinary 
topics 


63% 


8% 


50% 


79% 


94% 


Online reference resources 


62% 


22% 


52% 


71% 


90% 


Digital film or video 


62% 


16% 


57% 


73% 


82% 


Maps 


53% 


5% 


45% 


62% 


84% 


Online or digitized documents 


50% 


4% 


36% 


61% 


89% 


Audio materials 


46% 


9% 


38% 


53% 


70% 


Curricular materials and websites that are created by other faculty 
and/or other institutions 


35% 


3% 


26% 


40% 


64% 


Digital readers or coursepacks 


30% 


7% 


19% 


36% 


52% 


Online class discussions 


28% 


1% 


19% 


30% 


60% 


Government documents 


27% 


5% 


15% 


34% 


54% 


Data archives 


27% 


4% 


18% 


29% 


51% 


Digital facsimiles of ancient or historical manuscripts 


23% 


2% 


15% 


25% 


48% 


Simulations or animations 


19% 


2% 


8% 


17% 


53% 


Personal online diaries 


9% 


0% 


5% 


8% 


22% 


How often do you use digital resources in your undergraduate 
teaching from each of the following sources? 












Search engines/directories 


81% 


48% 


77% 


88% 


94% 


My own personal collection of digital materials 


69% 


29% 


59% 


81% 


92% 


Online journals 


62% 


45% 


52% 


67% 


83% 


Public online image databases 


62% 


29% 


51% 


69% 


88% 


Media sites 


57% 


22% 


50% 


63% 


80% 


Library collections 


57% 


29% 


47% 


63% 


85% 


“Portals” that provide links or URL’s relevant to particular disciplinary 
topics 


55% 


14% 


41% 


66% 


83% 


Online exhibits 


37% 


4% 


21% 


44% 


76% 


Campus image databases from my own institution 


24% 


4% 


17% 


27% 


42% 


Commercial image databases 


9% 


1% 


5% 


7% 


22% 


How often have you heard about sources of digital resources 
from each of the following? 












Word of mouth from colleagues 


74% 


41% 


73% 


79% 


83% 


Word of mouth from students 


57% 


29% 


51% 


63% 


71% 


Professional societies or discussion lists 


55% 


43% 


47% 


62% 


68% 


Recommendation from a campus librarian 


51% 


37% 


49% 


55% 


59% 


A campus department devoted to instructional technology 


41% 


16% 


39% 


45% 


53% 


How often do you use digital resources in each of these ways? 












Presented during my lectures/class 


71% 


22% 


63% 


83% 


94% 


Assigned to students for review and/or study 


59% 


14% 


53% 


66% 


83% 


Assigned for student research projects or problem-based learning 
assignments 


56% 


16% 


47% 


64% 


82% 


Posted directly on my course website 


52% 


15% 


41% 


60% 


79% 


Linked from my course website 


49% 


13% 


37% 


58% 


77% 


Used in tests and quizzes 


25% 


5% 


12% 


32% 


48% 


Assigned to students to create their own digital portfolios and/or 
multimedia projects 


20% 


0% 


10% 


22% 


46% 


Presented in the context of an online discussion 


12% 


0% 


5% 


13% 


32% 


Presented in my online lectures 


12% 


1% 


4% 


13% 


31% 
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All 


non- 

light 

users 


light- 

medium 

users 


medium- 

heavy 

users 


heavy 

users 


How much do you agree or disagree with the following 
statements about your reasons for using digital resources? 1 
use digital resources in my teaching... 












because it improves my students’ learning. 


78% 


31% 


75% 


84% 


96% 


to integrate primary source material into the course. 


78% 


49% 


68% 


86% 


98% 


to provide students a context tor a topic. 


75% 


35% 


70% 


80% 


95% 


to get students excited about a topic. 


73% 


35% 


68% 


79% 


92% 


because it allows me to do things in the classroom that 1 could never 
do otherwise. 


68% 


35% 


55% 


77% 


91% 


because it provides access to resources that we don’t have at our 
college. 


61% 


22% 


51% 


69% 


82% 


because it allows my students to be more creative. 


56% 


19% 


48% 


61% 


82% 


to teach critical thinking skills. 


56% 


20% 


47% 


63% 


78% 


to integrate my research interests into my course. 


55% 


25% 


45% 


64% 


75% 


because 1 like or feel very comfortable with the new technologies. 


53% 


12% 


42% 


64% 


78% 


because it is more convenient for my students and their schedules. 


51% 


25% 


46% 


53% 


68% 


to teach information literacy (i.e., evaluating the online materials 
themselves). 


43% 


10% 


32% 


50% 


69% 


because it saves me time. 


41% 


30% 


41% 


39% 


50% 


because my students expect or ask for more technology. 


38% 


13% 


33% 


43% 


49% 


because it allows me to stay up-to-date with my colleagues. 


37% 


11% 


32% 


43% 


50% 


to provide students with both good and bad examples of different 
kinds of scholarship. 


32% 


12% 


25% 


31% 


59% 


because it creates a sense of community for students enrolled in my 
course. 


32% 


5% 


24% 


34% 


55% 


because 1 enjoy having my teaching practices and course materials 
available to anyone in the world who would like to use them. 


25% 


10% 


13% 


28% 


50% 


because the administration encourages me to use digital resources 
more. 


23% 


10% 


22% 


25% 


28% 


to provide students a preview of the course before they register. 


22% 


10% 


16% 


20% 


42% 


because it may help me get promoted or get tenure. 


9% 


6% 


8% 


9% 


13% 


How much do you agree or disagree with the following 
statements about your reasons for nofusing digital resources? 
1 don’t use digital resources in certain teaching situations, 
because... 












they cannot substitute for the teaching approaches 1 use. 


75% 


86% 


81% 


74% 


59% 


1 don’t have time to use digital resources. 


66% 


75% 


67% 


67% 


55% 


using them distracts from the core goals of my teaching. 


47% 


61% 


51% 


45% 


31% 


1 don’t want my students to copy or plagiarize material from the web. 


33% 


44% 


37% 


29% 


23% 


they are irrelevant to my field. 


30% 


44% 


35% 


27% 


20% 


students don’t have the information literacy skills to assess the 
credibility of digital resources. 


29% 


34% 


30% 


30% 


20% 


digital material can be presented outside its original context. 


25% 


37% 


25% 


25% 


15% 


How much do you agree or disagree with the following 
statements? 1 have difficulty using digital resources the way 1 
would like, because... 












1 don’t have reliable access to physical resources in my classroom(s). 


53% 


49% 


56% 


60% 


40% 


the digital resources are distributed in so many places that it is 
difficult for me to organize them for use in my teaching. 


45% 


53% 


53% 


43% 


28% 


there are too many resources out there for me to take advantage 
of — 1 am overwhelmed. 


44% 


51% 


54% 


43% 


21% 


1 don’t have time to assess the credibility of the available resources. 


43% 


55% 


47% 


47% 


22% 


the content 1 need or want is just not available online. 


41% 


47% 


47% 


41% 


26% 


the academic quality of available materials is too poor to meet my 
needs. 


39% 


50% 


41% 


39% 


30% 


1 don’t have reliable access to scanners. 


39% 


38% 


41% 


43% 


28% 


1 don’t know how to locate the online materials 1 need. 


36% 


49% 


45% 


30% 


19% 
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All 


non- 

light 

users 


light- 

medium 

users 


medium- 

heavy 

users 


heavy 

users 


1 don’t know how to save presentations to my computer so they can 
be run without a live connection. 


35% 


39% 


39% 


38% 


17% 


available software is unsuitable for integrating audio or video into my 
course. 


34% 


20% 


35% 


38% 


34% 


search engines provide irrelevant results for my needs. 


34% 


38% 


41% 


30% 


22% 


1 have difficulty understanding the issues surrounding copyright and 
digital collections. 


33% 


34% 


36% 


33% 


27% 


my students don't have a high-speed connection. 


32% 


24% 


29% 


38% 


34% 


course management software packages are inadequate for my 
needs. 


32% 


29% 


33% 


35% 


27% 


web sites 1 would use are unreliable, and 1 can’t count on them being 
there when 1 need them. 


32% 


33% 


33% 


35% 


24% 


available software is unsuitable for viewing and displaying digital 
images. 


31% 


24% 


33% 


36% 


22% 


my students don’t have reliable access to computers. 


30% 


23% 


29% 


34% 


26% 


web formats allow me to link to whole documents, but not to specific 
excerpts within a text. 


28% 


27% 


31% 


32% 


17% 


it is difficult to get server space or access to a server in order to 
store/host digital resources for teaching. 


27% 


26% 


23% 


34% 


21% 


1 don’t have reliable access to a high-speed connection. 


21% 


29% 


24% 


21% 


10% 


1 don’t have reliable access to a computer. 


13% 


17% 


15% 


13% 


5% 


How important is it for you to have support or assistance with 
each of the following activities for your teaching? 












obtaining or setting up technical infrastructure 


82% 


70% 


80% 


88% 


84% 


creating my own website 


82% 


76% 


83% 


86% 


76% 


digitizing existing resources 


80% 


73% 


81% 


83% 


79% 


learning how to use a learning management system 


79% 


73% 


82% 


82% 


72% 


importing resources into a course website or a database 


79% 


77% 


82% 


80% 


71% 


gathering, organizing, and maintaining digital materials 


78% 


70% 


79% 


81% 


76% 


integrating resources into a learning management system 


78% 


70% 


79% 


80% 


75% 


finding digital resources 


72% 


65% 


77% 


73% 


66% 


training students to find or evaluate digital resources 


71% 


64% 


69% 


76% 


70% 


interpreting copyright laws and/or securing copyright permission 


65% 


52% 


65% 


67% 


69% 


assessing the credibility of digital resources 


50% 


44% 


55% 


48% 


49% 


evaluating the appropriateness of resources for my teaching goals 


42% 


41% 


44% 


41% 


38% 


How satisfied or dissatisfied are you with the support you have 
received from the following sources? 












Campus librarians 


84% 


87% 


86% 


88% 


75% 


Friends or family 


83% 


76% 


81% 


85% 


87% 


Graduate students 


80% 


87% 


80% 


78% 


80% 


Campus educational technology or IT support staff 


73% 


61% 


73% 


74% 


75% 


Other faculty 


71% 


82% 


73% 


74% 


60% 


Undergraduate students 


70% 


76% 


67% 


66% 


78% 


My departmental or college staff 


66% 


69% 


71% 


64% 


61% 


Workshops 


60% 


60% 


53% 


68% 


57% 


Online help or guides 


47% 


41% 


37% 


51% 


54% 














Overall digital resource use (0-100) 


31 


7 


23 


37 


54 


Overall technology use (0=no usage; 16=max usage) 


14.5 


13.7 


14.3 


14.7 


15.2 














Principal Component Scores: 












What resources people use 












General purpose and reference materials 


0.00 


-0.81 


-0.28 


0.17 


0.63 


Images and audiovisual materials 


0.02 


-0.91 


-0.22 


0.13 


0.76 


Historical documents, maps, and primary sources 


-0.03 


-0.66 


-0.32 


0.16 


0.50 


Data, news/media, and governmental resources 


0.00 


-0.53 


-0.22 


0.10 


0.52 
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All 


non- 

light 

users 


light- 

medium 

users 


medium- 

heavy 

users 


heavy 

users 


Discussion and curricular materials 


0.00 


-0.22 


-0.26 


-0.07 


0.74 


How people use digital resources 












Student assignments 


0.01 


-0.70 


-0.21 


0.13 


0.65 


Web posting 


0.00 


-0.54 


-0.21 


0.16 


0.44 


Online courses 


0.00 


-0.06 


-0.18 


-0.08 


0.51 


In-class presentation 


0.01 


-0.83 


-0.25 


0.26 


0.56 


Motivations for use or non-use 












Pedagogical reasons 


0.03 


-1.19 


-0.22 


0.26 


0.67 


Expectations and reputation 


-0.05 


-0.29 


-0.06 


0.02 


-0.03 


Inappropriateness 


0.02 


0.39 


0.09 


0.01 


-0.30 


Concerns about students' interpretation and information literacy 


0.01 


0.24 


0.02 


0.02 


-0.15 


Time, convenience, and access 


-0.02 


-0.43 


-0.12 


0.02 


0.28 


Teaching information literacy and critical thinking 


0.03 


-0.37 


-0.16 


0.08 


0.52 


Making information publicly available 


0.02 


-0.14 


-0.13 


-0.02 


0.47 


Using free and publicly available materials 


-0.06 


-0.15 


-0.13 


0.03 


-0.04 


Barriers to use 












Finding resources 


0.02 


0.19 


0.22 


-0.02 


-0.41 


High-end multimedia equipment and software 


0.00 


-0.26 


-0.03 


0.15 


-0.02 


Personal equipment 


0.00 


0.18 


0.01 


0.07 


-0.26 


Resource availability or existence 


0.02 


0.10 


0.11 


0.01 


-0.21 


Student equipment 


-0.01 


-0.21 


-0.06 


0.05 


0.09 


Activities with which support is needed 












Technical activities 


0.00 


-0.07 


-0.02 


0.13 


-0.13 


Intellectual and content-based activities 


0.01 


-0.03 


0.08 


-0.04 


-0.03 



http:/ / digitalresourcestudy.berkeley.edu 



10-60 






Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Table H.5: Faculty survey responses (community colleges only), broken out by institution 
size 





All 

community 

colleges 


small 


medium 


large 


N 


206 


51 


82 


73 


How often do you use the following types of digital resources in your 
undergraduate teaching: 










Images or visual materials 


80% 


82% 


82% 


77% 


News or other media sources and archives 


81% 


86% 


78% 


81% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


66% 


69% 


71% 


59% 


Online reference resources 


72% 


75% 


70% 


71% 


Digital film or video 


72% 


72% 


76% 


69% 


Maps 


51% 


60% 


51% 


46% 


Online or digitized documents 


43% 


48% 


47% 


36% 


Audio materials 


54% 


51% 


59% 


52% 


Curricular materials and websites that are created by other faculty and/or 
other institutions 


43% 


51% 


41% 


40% 


Digital readers or coursepacks 


20% 


21% 


26% 


12% 


Online class discussions 


27% 


31% 


33% 


18% 


Government documents 


35% 


33% 


40% 


30% 


Data archives 


31% 


29% 


39% 


25% 


Digital facsimiles of ancient or historical manuscripts 


20% 


17% 


22% 


21% 


Simulations or animations 


29% 


25% 


36% 


25% 


Personal online diaries 


12% 


19% 


12% 


6% 


How often do you use digital resources in your undergraduate teaching 
from each of the following sources? 










Search engines/directories 


87% 


83% 


89% 


87% 


My own personal collection of digital materials 


71% 


71% 


68% 


72% 


Online journals 


46% 


38% 


54% 


44% 


Public online image databases 


62% 


57% 


71% 


56% 


Media sites 


69% 


72% 


77% 


57% 


Library collections 


52% 


43% 


59% 


51% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


58% 


56% 


68% 


48% 


Online exhibits 


37% 


35% 


45% 


30% 


Campus image databases from my own institution 


26% 


13% 


34% 


26% 


Commercial image databases 


15% 


4% 


22% 


16% 


How often have you heard about sources of digital resources from each 
of the following? 










Word of mouth from colleagues 


73% 


67% 


79% 


71% 


Word of mouth from students 


61% 


69% 


64% 


54% 


Professional societies or discussion lists 


46% 


45% 


32% 


61% 


Recommendation from a campus librarian 


57% 


54% 


57% 


60% 


A campus department devoted to instructional technology 


46% 


38% 


51% 


47% 


How often do you use digital resources in each of these ways? 










Presented during my lectures/class 


78% 


73% 


82% 


77% 


Assigned to students for review and/or study 


64% 


67% 


66% 


60% 


Assigned for student research projects or problem-based learning 
assignments 


65% 


65% 


68% 


61% 


Posted directly on my course website 


38% 


37% 


42% 


34% 


Linked from my course website 


39% 


41% 


46% 


29% 


Used in tests and quizzes 


37% 


33% 


37% 


39% 


Assigned to students to create their own digital portfolios and/or multimedia 
projects 


22% 


20% 


29% 


17% 


Presented in the context of an online discussion 


21% 


20% 


26% 


16% 


Presented in my online lectures 


21% 


24% 


24% 


16% 
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All 

community 

colleges 


small 


medium 


large 


How much do you agree or disagree with the following statements 
about your reasons for using digital resources? 1 use digital resources 
in my teaching... 










because it improves my students’ learning. 


86% 


82% 


89% 


87% 


to integrate primary source material into the course. 


82% 


80% 


85% 


80% 


to provide students a context tor a topic. 


81% 


73% 


86% 


80% 


to get students excited about a topic. 


84% 


84% 


83% 


84% 


because it allows me to do things in the classroom that 1 could never do 
otherwise. 


68% 


73% 


70% 


61% 


because it provides access to resources that we don't have at our college. 


76% 


73% 


80% 


75% 


because it allows my students to be more creative. 


72% 


62% 


69% 


82% 


to teach critical thinking skills. 


75% 


73% 


77% 


75% 


to integrate my research interests into my course. 


59% 


62% 


56% 


61% 


because 1 like or feel very comfortable with the new technologies. 


63% 


58% 


71% 


57% 


because it is more convenient for my students and their schedules. 


55% 


51% 


66% 


45% 


to teach information literacy (i.e., evaluating the online materials themselves). 


56% 


51% 


61% 


53% 


because it saves me time. 


41% 


43% 


51% 


29% 


because my students expect or ask for more technology. 


43% 


40% 


44% 


44% 


because it allows me to stay up-to-date with my colleagues. 


55% 


52% 


59% 


53% 


to provide students with both good and bad examples of different kinds of 
scholarship. 


49% 


50% 


53% 


42% 


because it creates a sense of community for students enrolled in my course. 


40% 


40% 


47% 


33% 


because 1 enjoy having my teaching practices and course materials available 
to anyone in the world who would like to use them. 


31% 


29% 


33% 


30% 


because the administration encourages me to use digital resources more. 


32% 


21% 


37% 


34% 


to provide students a preview of the course before they register. 


27% 


28% 


30% 


22% 


because it may help me get promoted or get tenure. 


12% 


12% 


16% 


8% 


How much do you agree or disagree with the following statements 
about your reasons for not using digital resources? 1 don’t use digital 
resources in certain teaching situations, because... 










they cannot substitute for the teaching approaches 1 use. 


66% 


72% 


53% 


77% 


1 don’t have time to use digital resources. 


61% 


61% 


51% 


74% 


using them distracts from the core goals of my teaching. 


34% 


40% 


25% 


41% 


1 don’t want my students to copy or plagiarize material from the web. 


34% 


39% 


29% 


38% 


they are irrelevant to my field. 


28% 


33% 


21% 


35% 


students don’t have the information literacy skills to assess the credibility of 
digital resources. 


39% 


41% 


33% 


45% 


digital material can be presented outside its original context. 


31% 


32% 


28% 


33% 


How much do you agree or disagree with the following statements? 1 
have difficulty using digital resources the way 1 would like, because... 










1 don’t have reliable access to physical resources in my classroom(s). 


63% 


52% 


60% 


74% 


the digital resources are distributed in so many places that it is difficult for me 
to organize them for use in my teaching. 


46% 


43% 


46% 


48% 


there are too many resources out there for me to take advantage of — 1 am 
overwhelmed. 


47% 


48% 


47% 


47% 


1 don’t have time to assess the credibility of the available resources. 


44% 


46% 


39% 


48% 


the content 1 need or want is just not available online. 


25% 


24% 


23% 


27% 


the academic quality of available materials is too poor to meet my needs. 


24% 


27% 


19% 


28% 


1 don’t have reliable access to scanners. 


39% 


31% 


36% 


51% 


1 don’t know how to locate the online materials 1 need. 


40% 


41% 


41% 


37% 


1 don’t know how to save presentations to my computer so they can be run 
without a live connection. 


36% 


33% 


39% 


33% 


available software is unsuitable for integrating audio or video into my course. 


40% 


39% 


39% 


41% 


search engines provide irrelevant results for my needs. 


39% 


36% 


36% 


45% 


1 have difficulty understanding the issues surrounding copyright and digital 
collections. 


34% 


38% 


32% 


32% 
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All 

community 

colleges 


small 


medium 


large 


my students don't have a high-speed connection. 


54% 


53% 


47% 


62% 


course management software packages are inadequate for my needs. 


29% 


26% 


24% 


38% 


web sites 1 would use are unreliable, and 1 can’t count on them being there 
when 1 need them. 


33% 


44% 


26% 


34% 


available software is unsuitable for viewing and displaying digital images. 


31% 


24% 


34% 


34% 


my students don’t have reliable access to computers. 


54% 


48% 


52% 


60% 


web formats allow me to link to whole documents, but not to specific excerpts 
within a text. 


32% 


20% 


33% 


40% 


it is difficult to get server space or access to a server in order to store/host 
digital resources for teaching. 


38% 


44% 


34% 


37% 


1 don’t have reliable access to a high-speed connection. 


32% 


31% 


29% 


36% 


1 don’t have reliable access to a computer. 


21% 


13% 


17% 


33% 


How important is it for you to have support or assistance with each of 
the following activities for your teaching? 










obtaining or setting up technical infrastructure 


84% 


82% 


79% 


91% 


creating my own website 


81% 


84% 


78% 


83% 


digitizing existing resources 


78% 


75% 


75% 


84% 


learning how to use a learning management system 


80% 


80% 


79% 


83% 


importing resources into a course website or a database 


77% 


77% 


76% 


78% 


gathering, organizing, and maintaining digital materials 


77% 


73% 


74% 


83% 


integrating resources into a learning management system 


81% 


80% 


79% 


85% 


finding digital resources 


78% 


73% 


79% 


81% 


training students to find or evaluate digital resources 


79% 


84% 


76% 


78% 


interpreting copyright laws and/or securing copyright permission 


70% 


70% 


71% 


68% 


assessing the credibility of digital resources 


69% 


68% 


68% 


71% 


evaluating the appropriateness of resources for my teaching goals 


60% 


61% 


54% 


66% 


How satisfied or dissatisfied are you with the support you have received 
from the following sources? 










Campus librarians 


84% 


77% 


81% 


92% 


Friends or family 


90% 


93% 


88% 


91% 


Graduate students 


75% 


80% 


100% 


60% 


Campus educational technology or IT support staff 


68% 


68% 


68% 


68% 


Other faculty 


74% 


76% 


76% 


70% 


Undergraduate students 


69% 


87% 


72% 


54% 


My departmental or college staff 


63% 


54% 


68% 


62% 


Workshops 


67% 


61% 


68% 


70% 


Online help or guides 


53% 


50% 


48% 


60% 












Overall digital resource use (0-100) 


32 


34 


34 


29 


Overall technology use (0=no usage; 16=max usage) 


13.6 


13.9 


13.7 


13.4 












Principal Component Scores: 










What resources people use 










General purpose and reference materials 


-0.02 


-0.03 


0.07 


-0.12 


Images and audiovisual materials 


0.57 


0.46 


0.71 


0.50 


Historical documents, maps, and primary sources 


-0.35 


-0.37 


-0.34 


-0.34 


Data, news/media, and governmental resources 


0.19 


0.34 


0.24 


0.02 


Discussion and curricular materials 


-0.06 


-0.04 


0.09 


-0.24 


How people use digital resources 










Student assignments 


0.17 


0.09 


0.24 


0.15 


Web posting 


-0.39 


-0.38 


-0.31 


-0.51 


Online courses 


0.41 


0.47 


0.51 


0.25 


In-class presentation 


0.20 


0.11 


0.26 


0.19 


Motivations for use or non-use 










Pedagogical reasons 


0.21 


0.10 


0.15 


0.35 
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All 

community 

colleges 


small 


medium 


large 


Expectations and reputation 


0.12 


0.14 


0.13 


0.09 


Inappropriateness 


-0.33 


-0.36 


-0.57 


-0.03 


Concerns about students’ interpretation and information literacy 


0.25 


0.25 


0.18 


0.32 


Time, convenience, and access 


0.01 


0.05 


0.17 


-0.20 


Teaching information literacy and critical thinking 


0.40 


0.36 


0.45 


0.39 


Making information publicly available 


-0.04 


-0.05 


0.06 


-0.15 


Using free and publicly available materials 


-0.16 


0.00 


-0.10 


-0.34 


Barriers to use 










Finding resources 


0.05 


0.15 


0.04 


-0.03 


High-end multimedia equipment and software 


0.01 


-0.14 


-0.04 


0.19 


Personal equipment 


0.28 


0.18 


0.22 


0.42 


Resource availability or existence 


-0.29 


-0.29 


-0.30 


-0.27 


Student equipment 


0.54 


0.55 


0.46 


0.63 


Activities with which support is needed 










Technical activities 


-0.15 


-0.22 


-0.23 


-0.01 


Intellectual and content-based activities 


0.36 


0.32 


0.30 


0.46 
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Table H.6: Faculty survey responses (community colleges only), broken out by district 
population density 





All 

community 

colleges 


rural 


suburban 


urban 


N 


206 


87 


64 


55 


How often do you use the following types of digital resources in your 
undergraduate teaching: 










Images or visual materials 


80% 


78% 


87% 


76% 


News or other media sources and archives 


81% 


86% 


76% 


79% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


66% 


72% 


64% 


60% 


Online reference resources 


72% 


77% 


71% 


63% 


Digital film or video 


72% 


72% 


80% 


63% 


Maps 


51% 


57% 


46% 


49% 


Online or digitized documents 


43% 


49% 


47% 


31% 


Audio materials 


54% 


61% 


60% 


38% 


Curricular materials and websites that are created by other faculty and/or 
other institutions 


43% 


48% 


47% 


33% 


Digital readers or coursepacks 


20% 


18% 


23% 


19% 


Online class discussions 


27% 


22% 


40% 


19% 


Government documents 


35% 


40% 


26% 


35% 


Data archives 


31% 


35% 


22% 


35% 


Digital facsimiles of ancient or historical manuscripts 


20% 


19% 


22% 


20% 


Simulations or animations 


29% 


23% 


33% 


33% 


Personal online diaries 


12% 


16% 


9% 


8% 


How often do you use digital resources in your undergraduate teaching 
from each of the following sources? 










Search engines/directories 


87% 


89% 


85% 


87% 


My own personal collection of digital materials 


71% 


76% 


73% 


60% 


Online journals 


46% 


45% 


50% 


44% 


Public online image databases 


62% 


68% 


58% 


57% 


Media sites 


69% 


78% 


64% 


60% 


Library collections 


52% 


53% 


62% 


38% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


58% 


61% 


59% 


52% 


Online exhibits 


37% 


37% 


39% 


35% 


Campus image databases from my own institution 


26% 


27% 


27% 


22% 


Commercial image databases 


15% 


13% 


14% 


20% 


How often have you heard about sources of digital resources from 
each of the following? 










Word of mouth from colleagues 


73% 


75% 


74% 


68% 


Word of mouth from students 


61% 


60% 


68% 


56% 


Professional societies or discussion lists 


46% 


45% 


47% 


45% 


Recommendation from a campus librarian 


57% 


60% 


68% 


42% 


A campus department devoted to instructional technology 


46% 


56% 


47% 


29% 


How often do you use digital resources in each of these ways? 










Presented during my lectures/class 


78% 


80% 


78% 


75% 


Assigned to students for review and/or study 


64% 


67% 


74% 


50% 


Assigned for student research projects or problem-based learning 
assignments 


65% 


64% 


78% 


53% 


Posted directly on my course website 


38% 


33% 


58% 


24% 


Linked from my course website 


39% 


36% 


53% 


29% 


Used in tests and quizzes 


37% 


41% 


40% 


27% 


Assigned to students to create their own digital portfolios and/or multimedia 
projects 


22% 


22% 


25% 


20% 


Presented in the context of an online discussion 


21% 


16% 


34% 


14% 


Presented in my online lectures 


21% 


15% 


37% 


14% 
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All 

community 

colleges 


rural 


suburban 


urban 


How much do you agree or disagree with the following statements 
about your reasons for using digital resources? 1 use digital resources 
in my teaching... 










because it improves my students’ learning. 


86% 


88% 


87% 


84% 


to integrate primary source material into the course. 


82% 


85% 


85% 


76% 


to provide students a context tor a topic. 


81% 


83% 


79% 


78% 


to get students excited about a topic. 


84% 


85% 


83% 


82% 


because it allows me to do things in the classroom that 1 could never do 
otherwise. 


68% 


62% 


74% 


69% 


because it provides access to resources that we don't have at our college. 


76% 


82% 


74% 


70% 


because it allows my students to be more creative. 


72% 


69% 


72% 


76% 


to teach critical thinking skills. 


75% 


72% 


85% 


70% 


to integrate my research interests into my course. 


59% 


66% 


65% 


44% 


because 1 like or feel very comfortable with the new technologies. 


63% 


64% 


65% 


60% 


because it is more convenient for my students and their schedules. 


55% 


53% 


65% 


47% 


to teach information literacy (i.e., evaluating the online materials 
themselves). 


56% 


62% 


58% 


45% 


because it saves me time. 


41% 


46% 


41% 


35% 


because my students expect or ask for more technology. 


43% 


44% 


50% 


33% 


because it allows me to stay up-to-date with my colleagues. 


55% 


52% 


57% 


58% 


to provide students with both good and bad examples of different kinds of 
scholarship. 


49% 


55% 


43% 


45% 


because it creates a sense of community for students enrolled in my course. 


40% 


33% 


58% 


31% 


because 1 enjoy having my teaching practices and course materials available 
to anyone in the world who would like to use them. 


31% 


32% 


33% 


27% 


because the administration encourages me to use digital resources more. 


32% 


32% 


39% 


24% 


to provide students a preview of the course before they register. 


27% 


28% 


37% 


16% 


because it may help me get promoted or get tenure. 


12% 


13% 


13% 


10% 


How much do you agree or disagree with the following statements 
about your reasons for not using digital resources? 1 don’t use digital 
resources in certain teaching situations, because... 










they cannot substitute for the teaching approaches 1 use. 


66% 


70% 


64% 


62% 


1 don’t have time to use digital resources. 


61% 


66% 


66% 


48% 


using them distracts from the core goals of my teaching. 


34% 


34% 


36% 


34% 


1 don’t want my students to copy or plagiarize material from the web. 


34% 


29% 


36% 


40% 


they are irrelevant to my field. 


28% 


35% 


29% 


18% 


students don’t have the information literacy skills to assess the credibility of 
digital resources. 


39% 


36% 


27% 


57% 


digital material can be presented outside its original context. 


31% 


29% 


33% 


31% 


How much do you agree or disagree with the following statements? 1 
have difficulty using digital resources the way 1 would like, because... 










1 don’t have reliable access to physical resources in my classroom(s). 


63% 


62% 


54% 


74% 


the digital resources are distributed in so many places that it is difficult for 
me to organize them for use in my teaching. 


46% 


47% 


48% 


42% 


there are too many resources out there for me to take advantage of — 1 am 
overwhelmed. 


47% 


53% 


45% 


42% 


1 don’t have time to assess the credibility of the available resources. 


44% 


45% 


34% 


54% 


the content 1 need or want is just not available online. 


25% 


27% 


24% 


22% 


the academic quality of available materials is too poor to meet my needs. 


24% 


22% 


24% 


28% 


1 don’t have reliable access to scanners. 


39% 


41% 


33% 


44% 


1 don’t know how to locate the online materials 1 need. 


40% 


46% 


29% 


42% 


1 don’t know how to save presentations to my computer so they can be run 
without a live connection. 


36% 


42% 


27% 


36% 


available software is unsuitable for integrating audio or video into my course. 


40% 


46% 


36% 


34% 


search engines provide irrelevant results for my needs. 


39% 


42% 


38% 


36% 


1 have difficulty understanding the issues surrounding copyright and digital 
collections. 


34% 


35% 


29% 


38% 
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All 

community 

colleges 


rural 


suburban 


urban 


my students don't have a high-speed connection. 


54% 


59% 


38% 


64% 


course management software packages are inadequate for my needs. 


29% 


26% 


24% 


41% 


web sites 1 would use are unreliable, and 1 can’t count on them being there 
when 1 need them. 


33% 


44% 


25% 


27% 


available software is unsuitable for viewing and displaying digital images. 


31% 


37% 


28% 


28% 


my students don’t have reliable access to computers. 


54% 


55% 


36% 


72% 


web formats allow me to link to whole documents, but not to specific 
excerpts within a text. 


32% 


31% 


26% 


40% 


it is difficult to get server space or access to a server in order to store/host 
digital resources for teaching. 


38% 


39% 


28% 


47% 


1 don’t have reliable access to a high-speed connection. 


32% 


41% 


18% 


35% 


1 don’t have reliable access to a computer. 


21% 


27% 


4% 


32% 


How important is it for you to have support or assistance with each of 
the following activities for your teaching? 










obtaining or setting up technical infrastructure 


84% 


81% 


81% 


90% 


creating my own website 


81% 


82% 


78% 


84% 


digitizing existing resources 


78% 


80% 


69% 


86% 


learning how to use a learning management system 


80% 


79% 


78% 


85% 


importing resources into a course website or a database 


77% 


77% 


70% 


84% 


gathering, organizing, and maintaining digital materials 


77% 


75% 


72% 


84% 


integrating resources into a learning management system 


81% 


82% 


75% 


86% 


finding digital resources 


78% 


77% 


76% 


82% 


training students to find or evaluate digital resources 


79% 


78% 


74% 


86% 


interpreting copyright laws and/or securing copyright permission 


70% 


65% 


74% 


72% 


assessing the credibility of digital resources 


69% 


65% 


65% 


80% 


evaluating the appropriateness of resources for my teaching goals 


60% 


54% 


56% 


72% 


How satisfied or dissatisfied are you with the support you have 
received from the following sources? 










Campus librarians 


84% 


93% 


86% 


66% 


Friends or family 


90% 


94% 


89% 


88% 


Graduate students 


75% 


70% 


80% 


80% 


Campus educational technology or IT support staff 


68% 


71% 


80% 


49% 


Other faculty 


74% 


78% 


76% 


65% 


Undergraduate students 


69% 


54% 


79% 


75% 


My departmental or college staff 


63% 


66% 


64% 


55% 


Workshops 


67% 


64% 


79% 


56% 


Online help or guides 


53% 


42% 


69% 


50% 












Overall digital resource use (0-100) 


32 


33 


33 


30 


Overall technology use (0=no usage; 16=max usage) 


13.6 


13.6 


14.1 


13.3 












Principal Component Scores: 










What resources people use 










General purpose and reference materials 


-0.02 


0.04 


0.10 


-0.25 


Images and audiovisual materials 


0.57 


0.58 


0.62 


0.49 


Historical documents, maps, and primary sources 


-0.35 


-0.37 


-0.27 


-0.38 


Data, news/media, and governmental resources 


0.19 


0.38 


-0.07 


0.18 


Discussion and curricular materials 


-0.06 


-0.16 


0.18 


-0.19 


How people use digital resources 










Student assignments 


0.17 


0.15 


0.35 


0.01 


Web posting 


-0.39 


-0.41 


-0.20 


-0.58 


Online courses 


0.41 


0.27 


0.91 


0.10 


In-class presentation 


0.20 


0.23 


0.21 


0.14 


Motivations for use or non-use 










Pedagogical reasons 


0.21 


0.20 


0.29 


0.14 



http:/ / digitalresourcestudy.berkeley.edu 



10-67 






Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 





All 

community 

colleges 


rural 


suburban 


urban 


Expectations and reputation 


0.12 


0.05 


0.37 


-0.07 


Inappropriateness 


-0.33 


-0.30 


-0.25 


-0.46 


Concerns about students’ interpretation and information literacy 


0.25 


0.17 


0.13 


0.49 


Time, convenience, and access 


0.01 


0.02 


0.12 


-0.13 


Teaching information literacy and critical thinking 


0.40 


0.47 


0.36 


0.35 


Making information publicly available 


-0.04 


-0.07 


0.12 


-0.17 


Using free and publicly available materials 


-0.16 


-0.04 


-0.31 


-0.17 


Barriers to use 










Finding resources 


0.05 


0.15 


-0.03 


-0.03 


High-end multimedia equipment and software 


0.01 


-0.02 


-0.06 


0.14 


Personal equipment 


0.28 


0.38 


-0.10 


0.55 


Resource availability or existence 


-0.29 


-0.30 


-0.29 


-0.27 


Student equipment 


0.54 


0.52 


0.27 


0.88 


Activities with which support is needed 










Technical activities 


-0.15 


-0.19 


-0.19 


-0.07 


Intellectual and content-based activities 


0.36 


0.27 


0.29 


0.57 
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Table H.7: Faculty survey responses and H-Net survey responses 





California 

faculty 

survey 


H-Net 

Survey 


N 


831 


452 


How often do you use the following types of digital resources in your undergraduate 
teaching: 






Images or visual materials 


75% 


87% 


News or other media sources and archives 


64% 


71% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


63% 


70% 


Online reference resources 


62% 


62% 


Digital film or video 


62% 


62% 


Maps 


53% 


70% 


Online or digitized documents 


50% 


70% 


Audio materials 


46% 


47% 


Curricular materials and websites that are created by other faculty and/or other institutions 


35% 


45% 


Digital readers or coursepacks 


30% 


32% 


Online class discussions 


28% 


37% 


Government documents 


27% 


50% 


Data archives 


27% 


39% 


Digital facsimiles of ancient or historical manuscripts 


23% 


43% 


Simulations or animations 


19% 


22% 


Personal online diaries 


9% 


13% 


How often do you use digital resources in your undergraduate teaching from each of 
the following sources? 






Search engines/directories 


81% 


81% 


My own personal collection of digital materials 


69% 




My personal collection of digital resources that includes materials created by others 




77% 


My personal collection of digital resources that includes materials 1 have created myself 




71% 


A colleague’s personal collection of digital resources 




30% 


Online journals 


62% 


79% 


Public online image databases 


62% 


73% 


Media sites 


57% 


61% 


Library collections 


57% 


71% 


“Portals” that provide links or URL’s relevant to particular disciplinary topics 


55% 


60% 


Online exhibits 


37% 


52% 


Campus image databases from my own institution 


24% 


26% 


Commercial image databases 


9% 


12% 


How often have you heard about sources of digital resources from each of the 
following? 






Word of mouth from colleagues 


74% 


77% 


Word of mouth from students 


57% 


46% 


Professional societies or discussion lists 


55% 


84% 


Recommendation from a campus librarian 


51% 


58% 


A campus department devoted to instructional technology 


41% 


40% 


How often do you use digital resources in each of these ways? 






Presented during my lectures/class 


71% 


86% 


Assigned to students for review and/or study 


59% 


70% 


Assigned for student research projects or problem-based learning assignments 


56% 


71% 


Posted directly on my course website 


52% 


61% 


Linked from my course website 


49% 


63% 


Used in tests and quizzes 


25% 


38% 


Assigned to students to create their own digital portfolios and/or multimedia projects 


20% 


21% 


Presented in the context of an online discussion 


12% 


20% 


Presented in my online lectures 


12% 


16% 


How much do you agree or disagree with the following statements about your 
reasons for using digital resources? 1 use digital resources in my teaching... 
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California 

faculty 

survey 


H-Net 

Survey 


because it improves my students’ learning. 


78% 


88% 


to integrate primary source material into the course. 


78% 


86% 


to provide students a context for a topic. 


75% 


87% 


to get students excited about a topic. 


73% 


86% 


because it allows me to do things in the classroom that 1 could never do otherwise. 


68% 


84% 


because it provides access to resources that we don’t have at our college. 


61% 


76% 


because it allows my students to be more creative. 


56% 


62% 


to teach critical thinking skills. 


56% 


73% 


to integrate my research interests into my course. 


55% 


62% 


because 1 like or feel very comfortable with the new technologies. 


53% 


74% 


because it is more convenient for my students and their schedules. 


51% 


52% 


to teach information literacy (i.e., evaluating the online materials themselves). 


43% 


59% 


because it saves me time. 


41% 


42% 


because my students expect or ask for more technology. 


38% 


45% 


because it allows me to stay up-to-date with my colleagues. 


37% 


40% 


to provide students with both good and bad examples of different kinds of scholarship. 


32% 


40% 


because it creates a sense of community for students enrolled in my course. 


32% 


37% 


because 1 enjoy having my teaching practices and course materials available to anyone in 
the world who would like to use them. 


25% 


35% 


because the administration encourages me to use digital resources more. 


23% 


32% 


to provide students a preview of the course before they register. 


22% 


18% 


because it may help me get promoted or get tenure. 


9% 


20% 


How much do you agree or disagree with the following statements about your 
reasons for not using digital resources? 1 don’t use digital resources in certain 
teaching situations, because... 






they cannot substitute for the teaching approaches 1 use. 


75% 


70% 


1 don’t have time to use digital resources. 


66% 


63% 


using them distracts from the core goals of my teaching. 


47% 


45% 


1 don’t want my students to copy or plagiarize material from the web. 


33% 


30% 


they are irrelevant to my field. 


30% 


23% 


students don’t have the information literacy skills to assess the credibility of digital resources. 


29% 


25% 


digital material can be presented outside its original context. 


25% 


25% 


How much do you agree or disagree with the following statements? 1 have difficulty 
using digital resources the way 1 would like, because... 






1 don’t have reliable access to physical resources in my classroom(s). 


53% 


49% 


the digital resources are distributed in so many places that it is difficult for me to organize 
them for use in my teaching. 


45% 


42% 


there are too many resources out there for me to take advantage of — 1 am overwhelmed. 


44% 


42% 


1 don’t have time to assess the credibility of the available resources. 


43% 


41% 


the content 1 need or want is just not available online. 


41% 


48% 


the academic quality of available materials is too poor to meet my needs. 


39% 


41% 


1 don’t have reliable access to scanners. 


39% 


30% 


1 don’t know how to locate the online materials 1 need. 


36% 


30% 


1 don’t know how to save presentations to my computer so they can be run without a live 
connection. 


35% 


21% 


available software is unsuitable for integrating audio or video into my course. 


34% 


32% 


search engines provide irrelevant results for my needs. 


34% 


33% 


1 have difficulty understanding the issues surrounding copyright and digital collections. 


33% 


34% 


my students don't have a high-speed connection. 


32% 


35% 


course management software packages are inadequate for my needs. 


32% 


38% 


web sites 1 would use are unreliable, and 1 can’t count on them being there when 1 need 
them. 


32% 


33% 


available software is unsuitable for viewing and displaying digital images. 


31% 


29% 


my students don’t have reliable access to computers. 


30% 


32% 


web formats allow me to link to whole documents, but not to specific excerpts within a text. 


28% 


26% 
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California 

faculty 

survey 


H-Net 

Survey 


it is difficult to get server space or access to a server in order to store/host digital resources 
for teaching. 


27% 


28% 


1 don’t have reliable access to a high-speed connection. 


21% 


15% 


1 don’t have reliable access to a computer. 


13% 


11% 


How important is it for you to have support or assistance with each of the following 
activities for your teaching? 






obtaining or setting up technical infrastructure 


82% 


86% 


creating my own website 


82% 


75% 


digitizing existing resources 


80% 


75% 


learning how to use a learning management system 


79% 


74% 


importing resources into a course website or a database 


79% 


70% 


gathering, organizing, and maintaining digital materials 


78% 


71% 


integrating resources into a learning management system 


78% 


76% 


finding digital resources 


72% 


63% 


training students to find or evaluate digital resources 


71% 


65% 


interpreting copyright laws and/or securing copyright permission 


65% 


69% 


assessing the credibility of digital resources 


50% 


42% 


evaluating the appropriateness of resources for my teaching goals 


42% 


38% 


How satisfied or dissatisfied are you with the support you have received from the 
following sources? 






Campus librarians 


84% 


80% 


Friends or family 


83% 


84% 


Graduate students 


80% 


81% 


Campus educational technology or IT support staff 


73% 


70% 


Other faculty 


71% 


69% 


Undergraduate students 


70% 


67% 


My departmental or college staff 


66% 


61% 


Workshops 


60% 


57% 


Online help or guides 


47% 


55% 








Overall digital resource use (0-100) 


31 


35 


Overall technology use (0=no usage; 16=max usage) 


14.5 


15.4 
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Appendix I: Specific digital resources listed by faculty 

Table 1.1: Faculty survey results: What specific digital resources do you use? (categorized) 



Type of site 


# responses 


% responses 


# people 


% people 


Database/archive/ collection/ dig lib 


306 


11.7% 


219 


23.0% 


Scholarly journals 


231 


8.8% 


200 


21.0% 


Search engines 


185 


7.1% 


172 


18.1% 


Abstracting & indexing databases 


154 


5.9% 


122 


12.8% 


News/ Media/ Magazines 


169 


6.5% 


120 


12.6% 


Images & Video collections 


137 


5.2% 


115 


12.1% 


Government sites 


154 


5.9% 


107 


11.3% 


Library catalogues 


115 


4.4% 


102 


10.7% 


Curricular materials (incl. textbook resources) 


118 


4.5% 


89 


9.4% 


Other 


102 


3.9% 


88 


9.3% 


Portals 


83 


3.2% 


73 


7.7% 


My own collection 


75 


2.9% 


68 


7.2% 


Non-profit organizations / institutions 


79 


3.0% 


67 


7.0% 


Reference resources 


82 


3.1% 


61 


6.4% 


Museum websites (real, physical museums) 


73 


2.8% 


57 


6.0% 


Profession/ Academic Associations/ Societies 


57 


2.2% 


54 


5.7% 


Mailing lists, discussion groups, or blogs 


47 


1.8% 


36 


3.8% 


Commercial sites (corporations or retailers) 


28 


1.1% 


24 


2.5% 


Geography materials (maps, data, etc.) 


29 


1.1% 


21 


2.2% 


Bibliographies 


19 


0.7% 


18 


1.9% 


Social science (arch, anthro, poli sci) 


17 


0.6% 


17 


1.8% 


Arts (fine arts, performing, arch) 


16 


0.6% 


15 


1.6% 


Data collections (primarily numeric data) 


19 


0.7% 


15 


1.6% 


Audio collections 


8 


0.3% 


8 


0.8% 



Table 1.2: Faculty survey results: What specific digital resources do you use? 



Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


JSTOR - The Scholarly Journal Archive 


http : / / www . j stor . or g 


154 


5.9% 


153 


16.1% 


Google 


www.google.com 


142 


5.4% 


142 


14.9% 


Library of Congress 


loc.gov 


42 


1.6% 


42 


4.4% 


Library of Congress American Memory 
Project 


http:// memory.loc.gov/ 


42 


1.6% 


42 


4.4% 


FindLaw 




40 


1.5% 


39 


4.1% 


Lexis Nexis 


http : / / www .lexisnexis .com 


40 


1.5% 


40 


4.2% 


my own digitized images 




37 


1.4% 


37 


3.9% 


| libraries - library databases/ catalogues 


34 


1.3% 


34 


3.6% 


New York Times 


http: / /www .nytimes.com 


32 


1.2% 


32 


3.4% 


news/newspapers online 




30 


1.1% 


30 


3.2% 


websites - museums in general 




30 


1.1% 


30 


3.2% 


my own digital resources 




28 


1.1% 


28 


2.9% 


search engines 




25 


1.0% 


25 


2.6% 


Modern Language Association (MLA) 


http://www.mla.org/ 


23 


0.9% 


23 


2.4% 


| websites/ databases - images - general 


22 


0.8% 


22 


2.3% 


Perseus Digital Libraries 


http://www.perseus.tufts.edu 


21 


0.8% 


21 


2.2% 


Oxford English Dictionary online 


http://www.oed.com/ 


20 


0.8% 


20 


2.1% 


Project Muse 


http://muse.jhu.edu/ 


19 


0.7% 


19 


2.0% 


database - general 




18 


0.7% 


18 


1.9% 


websites - government 




18 


0.7% 


18 


1.9% 


websites - of textbook publisher 




17 


0.6% 


17 


1.8% 


Census 


http: / /www.census.gov 


15 


0.6% 


15 


1.6% 


Melvyl 


http: / /melvyl.cdlib.org/ 


15 


0.6% 


15 


1.6% 


list - H-Net 




15 


0.6% 


15 


1.6% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


library - general 




14 


0.5% 


14 


1.5% 


History Matters 


http:/ /history matters, gmu.edu 


13 


0.5% 


13 


1.4% 


Yahoo 


yahoo.com 


13 


0.5% 


13 


1.4% 


EEBO (early english books online) 


http: / /eebo. chadwyck.com/ 


12 


0.5% 


12 


1.3% 


California Digital Library (CDL) 


http: //www.cdlib.org/ 


11 


0.4% 


11 


1.2% 


EBSCO 


http://www.ebsco.com/home/ 


11 


0.4% 


11 


1.2% 


PBS 


http://www.pbs.org 


11 


0.4% 


11 


1.2% 


ProQuest 


http:/ / www.umi.com/ proquest 
/ 


11 


0.4% 


11 


1.2% 


Project Gutenberg 


http : / / gutenberg.net 


11 


0.4% 


11 


1.2% 


online collections of maps 




11 


0.4% 


11 


1.2% 


National Archives & Records 
Adminstration (NARA) 


http : / / www .archives .gov/ 


10 


0.4% 


10 


1.1% 


The Art Museum Image Consortium 
(AMICO) 


http://www.amico.org 


10 


0.4% 


10 


1.1% 


| database - Expanded Academic Indexes ASAP 


10 


0.4% 


10 


1.1% 


Internet Medieval Sourcebook 


http://www.fordham.edu/hals 
all/ sbookl j .html 


9 


0.3% 


9 


0.9% 


Internet Sourcebooks - by Paul Halsall 


http://www.fordham.edu/hals 

all/ 


9 


0.3% 


9 


0.9% 


database - InfoTrac 




9 


0.3% 


9 


0.9% 


FindLaw 


findlaw.com 


8 


0.3% 


8 


0.8% 


Healy's Literature Online (LION) 


http: / /lion.chadwyck. co.uk 


8 


0.3% 


8 


0.8% 


Labyrinth 


http:// www.georgetown.edu/1 
abyrinth/ 


8 


0.3% 


8 


0.8% 


Voice of the Shuttle 


http:// vos.ucsb.edu/ 


8 


0.3% 


8 


0.8% 


websites - film 




8 


0.3% 


8 


0.8% 


websites - links to primary sources 




8 


0.3% 


8 


0.8% 


Avalon Project 


http : / / www .yale.edu/ lawweb / 
avalon/ avalon. htm 


7 


0.3% 


7 


0.7% 


NPR National Public Radio 


http://www.npr.org/ 


7 


0.3% 


7 


0.7% 


OCLC (Online Computer Library Center) 
includes Worldcat and 


http:// www.oclc.org/ 


7 


0.3% 


7 


0.7% 


OWL 


http: / /owl.english.purdue.edu/ 


7 


0.3% 


7 


0.7% 


online portals 




7 


0.3% 


7 


0.7% 


Bedford/St. Martin's publisher - online 
companion 


http : / / www .bedf or dstmartins . c 
om/ 


6 


0.2% 


6 


0.6% 


Historical Maps (University of Texas at 
Austin) 


http://www.lib.utexas.edu/ma 

ps/ 


6 


0.2% 


6 


0.6% 


MERLOT (Multimedia Educational 
Resource for Learning and Onl 


http://www.merlot.org/Home. 

po 


6 


0.2% 


6 


0.6% 


Metropolitan Museum of Art 


http://www.metmuseum.org/ 


6 


0.2% 


6 


0.6% 


University of Virginia (UVA) Electronic 
Text 


http:/ / etext.lib.virginia.edu/ 


6 


0.2% 


6 


0.6% 


Washington Post archives 


http:/ /www.washingtonpost.co 
m/ 


6 


0.2% 


6 


0.6% 


libraries - UC library catalogues 




6 


0.2% 


6 


0.6% 


online dictionaries/ reference 




6 


0.2% 


6 


0.6% 


CNN 


CNN.com 


5 


0.2% 


5 


0.5% 


ERIC 


http://eric.ed.gov/ 


5 


0.2% 


5 


0.5% 


GALE Eighteenth Century Collections 
Online 


http:// www.gale.com/Eighteen 
thCentury/ 


5 


0.2% 


5 


0.5% 


Houghton Mifflin Company publisher's 
site 


http: //www.hmco.com/ 


5 


0.2% 


5 


0.5% 


ICPSR (InterUniversity Consortium for 
Political and Social R 


http:/ /www.icpsr.org/ 


5 


0.2% 


5 


0.5% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


Latin American Network Information 
Center 


http:/ /lanic.utexas.edu 


5 


0.2% 


5 


0.5% 


Mark Harden's Artchive 


http: //www.artchive.com/ 


5 


0.2% 


5 


0.5% 


Oyez 


http: / /www.oyez.org/ 


5 


0.2% 


5 


0.5% 


Smithsonian 


http: / / www.si.edu/ 


5 


0.2% 


5 


0.5% 


Thesaurus Linguae Graecae 


http: / / www.tlg.uci.edu/ 


5 


0.2% 


5 


0.5% 


Women and Social Movements 


http: / / womhist.binghamton.ed 

w 


5 


0.2% 


5 


0.5% 


database - America: History and Life 




5 


0.2% 


5 


0.5% 


database - Historical Abstracts 




5 


0.2% 


5 


0.5% 


database - online articles 




5 


0.2% 


5 


0.5% 


online bibliographies 




5 


0.2% 


5 


0.5% 


online exhibitions 




5 


0.2% 


5 


0.5% 


websites - for authors and issues 




5 


0.2% 


5 


0.5% 


websites - of artists 




5 


0.2% 


5 


0.5% 


websites - of other colleagues 




5 


0.2% 


5 


0.5% 


websites - syllabi of other teachers 




5 


0.2% 


5 


0.5% 


websites - university-based 




5 


0.2% 


5 


0.5% 


Archnet (Archeology Net) 


http://archnet.asu.edu/ 


4 


0.2% 


4 


0.4% 


Artcyclopedia 


http: / / www.artcyclopedia.com 
/ 


4 


0.2% 


4 


0.4% 


BBC 


http: / / www .bbc.co.uk/ 


4 


0.2% 


4 


0.4% 


Bartleby 


http:/ /www .bartleby .com 


4 


0.2% 


4 


0.4% 


Documenting the American South 


http://docsouth.unc.edu/neh 


4 


0.2% 


4 


0.4% 


Early Canadiana Online 


http:// www.canadiana.org/ eco 
/index.html 


4 


0.2% 


4 


0.4% 


Grove Dictionary of Art Online 


http : / / www . groveart . com/ inde 
x.html 


4 


0.2% 


4 


0.4% 


Internet Modem History Sourcebook 


http : / / www .f or dham. edu / hals 
all / mod / modsbook.html 


4 


0.2% 


4 


0.4% 


Making of America 


http://cdl.library.cornell.edU/m 

oa/ 


4 


0.2% 


4 


0.4% 


Middle English Compendium (includes 
Dictionary) 


http:/ /ets.umdl.umich.edu/m/ 
mec/ 


4 


0.2% 


4 


0.4% 


Miguel de Cervantes Digital Library 


http : / / www.cervantesvirtual.co 
m/ 


4 


0.2% 


4 


0.4% 


SPIRO 


http:/ /www.mip .berkeley.edu/ 
spiro/ 


4 


0.2% 


4 


0.4% 


THOMAS 


http:// thomas.loc.gov/ 


4 


0.2% 


4 


0.4% 


World Bank 


http://www.worldbank.org 


4 


0.2% 


4 


0.4% 


archive - electronic archives 




4 


0.2% 


4 


0.4% 


news online - French 




4 


0.2% 


4 


0.4% 


websites - historical 




4 


0.2% 


4 


0.4% 


websites - literature/ literary 




4 


0.2% 


4 


0.4% 


Arts & Letters Daily 


http: / /www.aldaily.com 


3 


0.1% 


3 


0.3% 


Bibliography of the History of Art 


The BHA is available via the 
Internet by subscription from D 


3 


0.1% 


3 


0.3% 


Bibliotheque nationale de France 


http://www.bnf.fr/ 


3 


0.1% 


3 


0.3% 


Blogs 




3 


0.1% 


3 


0.3% 


Central Intelligence Agency (CIA) 


http://www.cia.gov/ 


3 


0.1% 


3 


0.3% 


Cleveland Museum of Art Image 
Database 


http://www.clevelandart.org/ 


3 


0.1% 


3 


0.3% 


Corbis digital photo archive 


http:/ /www.corbis.com/ 


3 


0.1% 


3 


0.3% 


Encyclopedia Britannica online 


http://www.eb.com/ 


3 


0.1% 


3 


0.3% 


Great Buildings Collection 


Greatbuildings .com 


3 


0.1% 


3 


0.3% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


History Cooperative 


http:/ /www.historycooperative. 
org/ 


3 


0.1% 


3 


0.3% 


Instituto Cervantes 


http: / / www.cervantes.es/ 


3 


0.1% 


3 


0.3% 


Los Angeles Times (LA times) 


http://www.latimes.com/ 


3 


0.1% 


3 


0.3% 


Online Reference Book for Medieval 
Studies 


http: / / www.the-orb.net 


3 


0.1% 


3 


0.3% 


Orion (UCLA Library Catalogue) 


http: / /orion2.1ibrary. ucla.edu / 


3 


0.1% 


3 


0.3% 


Population Reference Bureau 


http: / /www.prb.org/ 


3 


0.1% 


3 


0.3% 


Prelinger Archives 


http: //www.archive.org/movie 
s/prelinger.php 


3 


0.1% 


3 


0.3% 


Real Academia Espanola 


http: / /www. rae.es/ 


3 


0.1% 


3 


0.3% 


Rhizome 


http: / /rhizome.org/ 


3 


0.1% 


3 


0.3% 


The Valley of Shadow 


http: //valley .vcdh.virginia.edu 
/ 


3 


0.1% 


3 


0.3% 


Thomson Heinle publisher 


http: / /www.heinle.com/ 


3 


0.1% 


3 


0.3% 


USGS 


http://www.usgs.gov/ 


3 


0.1% 


3 


0.3% 


Web of Science 


http:// www.isinet.com/produc 
ts/ citation/wos/ 


3 


0.1% 


3 


0.3% 


Webmuseum 


http: / /www.ibiblio.org/wm/ 


3 


0.1% 


3 


0.3% 


database - Anthropological Literature 




3 


0.1% 


3 


0.3% 


database - museum image collection 




3 


0.1% 


3 


0.3% 


libraries - UC Berkeley library electronic 
indexes 


http : / / www .lib .berkeley.edu/ 


3 


0.1% 


3 


0.3% 


libraries - UCLA collections 




3 


0.1% 


3 


0.3% 


libraries - universities 




3 


0.1% 


3 


0.3% 


online catalogues - general 




3 


0.1% 


3 


0.3% 


online discussion groups 




3 


0.1% 


3 


0.3% 


online media 




3 


0.1% 


3 


0.3% 


online periodicals 




3 


0.1% 


3 


0.3% 


| online resources - digital statistical resources 


3 


0.1% 


3 


0.3% 


| websites - audio 




3 


0.1% 


3 


0.3% 


(websites - links to archaeological material 


3 


0.1% 


3 


0.3% 


(ARTFL) American and French Research 
on the Treasury of the 


http://humanities.uchicago.edu 
/orgs/ ARTFL/ 


2 


0.1% 


2 


0.2% 


AP Photo Archive 


http:/ /ap. accuweather.com/app 
hoto/ 


2 


0.1% 


2 


0.2% 


ARTstor 


http://www.artstor.org/ 


2 


0.1% 


2 


0.2% 


ATLA (American Theological Library 
Association) Religion Dat 


http: / / www.atla.com/products 
/ catalogs/ catalogs_rdb.html 


2 


0.1% 


2 


0.2% 


Access to Insight 


http: / / www.accesstoinsight.org 
/ 


2 


0.1% 


2 


0.2% 


Adobe 


http://www.adobe.com/ 


2 


0.1% 


2 


0.2% 


AfricaFocus - University of Wisconsin, 
Madison 


http: / / webcat.library.wisc.edu:3 
200/ AfricaFocus/ 


2 


0.1% 


2 


0.2% 


Amazon 


Amazon.com 


2 


0.1% 


2 


0.2% 


American Anthropological Association 


http://www.aaanet.org/ 


2 


0.1% 


2 


0.2% 


American Museum of the Moving Image 


http: / /www.ammi.org 


2 


0.1% 


2 


0.2% 


American Sociological Association 


http://www.asanet.org/ 


2 


0.1% 


2 


0.2% 


American Studies Electronic Crossroad 
Project 


http:// www.georgetown.edu/ c 
rossroads/ 


2 


0.1% 


2 


0.2% 


American Studies, University of Virginia 


http: / /xroads.virginia.edu/ 


2 


0.1% 


2 


0.2% 


Annual Egyptological Bibliography 


http:/ /www.leidenuniv.nl/nino 
/aeb.html 


2 


0.1% 


2 


0.2% 


Apple 


Apple.com 


2 


0.1% 


2 


0.2% 


Archaelogical Data Serivce 


http://ads.ahds.ac.uk/ 


2 


0.1% 


2 


0.2% 


Art Index 


www.artindex.com 


2 


0.1% 


2 


0.2% 
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Avery library 


http://www.columbia.edU/cu/l 
web/ indiv/ avery / D Acollection 
s.ht 


2 


0.1% 


2 


0.2% 


Beazley 


http://www.beazley.ox.ac.uk/ 


2 


0.1% 


2 


0.2% 


Becoming Human 


http://www.becominghuman.o 

rg/ 


2 


0.1% 


2 


0.2% 


British Columbia Archives 


http : / / www .bcarchives . gov .be . c 
a/sn-14769EA/ index.htm 


2 


0.1% 


2 


0.2% 


Brown Women Writers Project 


http : / / www . wwp .brown, edu / 


2 


0.1% 


2 


0.2% 


C18-L (18th century studies) 


http:// www.personal.psu.edu/ 
special/ C18/cl8-l.htm 


2 


0.1% 


2 


0.2% 


CELT - Corpus of Electronic Texts 


http://www.ucc.ie/celt/ 


2 


0.1% 


2 


0.2% 


Canadian Broadcasting Corporation 


http://archives.cbc.ca/index.as 

p?IDLan=l 


2 


0.1% 


2 


0.2% 


Center for Disease Control and Prevention 


http: / / www.cdc.gov/ 


2 


0.1% 


2 


0.2% 


Center for History and New Media, 
George Mason University 


http : / / chnm. gmu.edu / indexl .h 
tml 


2 


0.1% 


2 


0.2% 


Chicago Historical Society 


http://www.chicagohs.org/ 


2 


0.1% 


2 


0.2% 


Chronicle of Higher Education 


http : / / chronicle.com / 


2 


0.1% 


2 


0.2% 


Congressional Quarterly Researcher 


http://library2.cqpress.com/ cqr 
esearcher/ 


2 


0.1% 


2 


0.2% 


Cornell University 


cornell.edu 


2 


0.1% 


2 


0.2% 


Digital Scriptorium 


http:/ /sunsite.berkeley.edu/Scr 
iptorium/ 


2 


0.1% 


2 


0.2% 


|ESTC The English Short Title Catalogue database through Melv 


2 


0.1% 


2 


0.2% 


Early American Imprints (Digital 
Collection) 


http:// www.readex.com/ digital 
/ digcoll.html 


2 


0.1% 


2 


0.2% 


Electronic Poetry Center (EPC) 


http://wings.buffalo.edu/epc/ 


2 


0.1% 


2 


0.2% 


Gallup (poll) 


http://www.gallup.com/poll/ 


2 


0.1% 


2 


0.2% 


Gilder Lehrman Center for the Study of 
Slavery, Resistance, 


http:// www.yale.edu/glc/ 


2 


0.1% 


2 


0.2% 


Guggenheim Museum 


http: / /www. guggenheim.org/ 


2 


0.1% 


2 


0.2% 


Guide to Grammar and Writing 


http://www.ccc.commnet.edu / 
grammar/ 


2 


0.1% 


2 


0.2% 


Harper's Magazine 


http://www.harpers.org/ 


2 


0.1% 


2 


0.2% 


Harvard Chaucer Page 


http://www.courses.fas.harvar 

d.edu/~chaucer/ 


2 


0.1% 


2 


0.2% 


Himalayan Art 


http: / / www. himalayanart.org/ 


2 


0.1% 


2 


0.2% 


Historical Census 


http: / / fisher.lib .virginia.edu/co 
llections / stats / histcensus / 


2 


0.1% 


2 


0.2% 


History Channel 


http: / /www.historychannel.co 
m/ 


2 


0.1% 


2 


0.2% 


IPUMS (Integrated Public Use Microdata 
Series) 


www.ipums.umn.edu 


2 


0.1% 


2 


0.2% 


International Medieval Bibliography 


http: / / www.brepols.net/publis 
hers/imb_info_en.htm 


2 


0.1% 


2 


0.2% 


Internet Classics Archive 


http: / / classics.mit.edu/ 


2 


0.1% 


2 


0.2% 


Internet History of Science Sourcebook 


http://www.fordham.edu/hals 

all/science/sciencesbook.html 


2 


0.1% 


2 


0.2% 


Kairos 


http://english.ttu.edu/kairos/ 


2 


0.1% 


2 


0.2% 


Lacus Curtius 


http://penelope.uchicago.edu/ 
Thayer / E/ Roman / home.html 


2 


0.1% 


2 


0.2% 


Literature Resource Center 


http: / /www. gale.com/LitRC/ 


2 


0.1% 


2 


0.2% 


MSNBC 


MSNBC.com 


2 


0.1% 


2 


0.2% 


McGraw-Hill publisher's site 


http://www.mcgraw-hill.com/ 


2 


0.1% 


2 


0.2% 


NASA 


nasa.gov 


2 


0.1% 


2 


0.2% 
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NCTE National Council of Teachers of 
English 


http: / / www.ncte.org/ 


2 


0.1% 


2 


0.2% 


National Archives of Canada 


http: / / www.collectionscanada.c 

a/ 


2 


0.1% 


2 


0.2% 


National Center for Science Education 


http://www.ncseweb.org/ 


2 


0.1% 


2 


0.2% 


National Election Studies 


http://www.umich.edu/~nes/ 


2 


0.1% 


2 


0.2% 


National Institute of Health 


http://www.nih.gov/ 


2 


0.1% 


2 


0.2% 


Netscape 


http: / /netscape.com 


2 


0.1% 


2 


0.2% 


New Testament Gateway 


http: / /www .ntgateway.com/ 


2 


0.1% 


2 


0.2% 


Paradigm Online Writing Assistant 
(POWA) 


http://powa.org/ 


2 


0.1% 


2 


0.2% 


Patrologia Latina 


http://pld.chadwyck.com/ 


2 


0.1% 


2 


0.2% 


Personal website of authors/literary 
movements 


http:// guweb2.gonzaga.edu/fac 
ulty/campbell/enl311/aufram.h 
tm 


2 


0.1% 


2 


0.2% 


Petrie Museum of Egyption Archaeology 


http://www.petrie.ucl.ac.uk/ 


2 


0.1% 


2 


0.2% 


Pew Research Center 


http: // people-press.org/ 


2 


0.1% 


2 


0.2% 


Polling Report.com 


pollingreport.com 


2 


0.1% 


2 


0.2% 


Portraits/Shaping of the Modern 
World/ Brooklyn College 


http: / / academic.brooklyn.cuny. 
edu/history/ virtual/ portrait, h 


2 


0.1% 


2 


0.2% 


Poynter Online (for Journalists) 


http:// poynter.org/ 


2 


0.1% 


2 


0.2% 


Quia 


http: / /www.quia.com/ 


2 


0.1% 


2 


0.2% 


Royal Historical Society Bibliography 


http://www.rhs.ac.uk/bibweLa 

sp 


2 


0.1% 


2 


0.2% 


SF Chronicle archives 


http : / / ww w .sf gate .com/ chroni 
cle/ 


2 


0.1% 


2 


0.2% 


The Academy American of Poets 


poets.org 


2 


0.1% 


2 


0.2% 


The Bryn Mawr Classical Review 


http://ccat.sas.upenn.edu/bmcr 

/ 


2 


0.1% 


2 


0.2% 


The Getty Collections 


http:/ /www. getty.edu/art/coll 
ections 


2 


0.1% 


2 


0.2% 


UBU Web 


http://www.ubu.com/ 


2 


0.1% 


2 


0.2% 


Victorian Web 


http:/ /www .victorianweb.org/ 


2 


0.1% 


2 


0.2% 


Wikipedia 


http://en.wikipedia.org/wiki/ 

Main_Page 


2 


0.1% 


2 


0.2% 


Xipolis 


http:/ /www.xipolis.net/ 


2 


0.1% 


2 


0.2% 


database - Ethnic News Watch 




2 


0.1% 


2 


0.2% 


database - Ingenta 


www.ingentaconnect.com 


2 


0.1% 


2 


0.2% 


database - PsycINFO 




2 


0.1% 


2 


0.2% 


database - Sociological Abstracts 




2 


0.1% 


2 


0.2% 


database - audio/ audio archives 




2 


0.1% 


2 


0.2% 


database - government 




2 


0.1% 


2 


0.2% 


images - art images 




2 


0.1% 


2 


0.2% 


libraries - Bancroft Library online 




2 


0.1% 


2 


0.2% 


libraries - Bodleian Library, Oxford 




2 


0.1% 


2 


0.2% 


| libraries - Cambridge University Library (UK) 


2 


0.1% 


2 


0.2% 


libraries - manuscript websites 




2 


0.1% 


2 


0.2% 


list - AnSax-L 




2 


0.1% 


2 


0.2% 


list- H-LATAM 




2 


0.1% 


2 


0.2% 


my own blog 




2 


0.1% 


2 


0.2% 


|my own department's digitized slide collection 


2 


0.1% 


2 


0.2% 


my own video 




2 


0.1% 


2 


0.2% 


news online - Indian 




2 


0.1% 


2 


0.2% 


news online - US 




2 


0.1% 


2 


0.2% 


online archaeology information 




2 


0.1% 


2 


0.2% 


online coursepacks 




2 


0.1% 


2 


0.2% 
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online data sources in general 




2 


0.1% 


2 


0.2% 


online indexes - for art history 




2 


0.1% 


2 


0.2% 


online resources - art history 




2 


0.1% 


2 


0.2% 


online resources - for writing 




2 


0.1% 


2 


0.2% 


(personal collection - a colleague's digital photos 


2 


0.1% 


2 


0.2% 


resources/ collections - French 




2 


0.1% 


2 


0.2% 


websites - ESL 




2 


0.1% 


2 


0.2% 


websites - US state departments sites 


http : / / www .state . gov / 


2 


0.1% 


2 


0.2% 


websites - companion for writing handboo 


k 


2 


0.1% 


2 


0.2% 


websites - grammar 




2 


0.1% 


2 


0.2% 


websites - music 




2 


0.1% 


2 


0.2% 


websites - poetry collections 




2 


0.1% 


2 


0.2% 


websites - reviews of books/ articles 




2 


0.1% 


2 


0.2% 


(websites - with government publications 


2 


0.1% 


2 


0.2% 


A AT A Online 


http : / / aata . getty .edu/ NPS/ 


1 


0.0% 


1 


0.1% 


AATG American Association of Teachers 
of German 


http:// www.aatg.org/ 


1 


0.0% 


1 


0.1% 


ABZU Bibliography 


http://www.etana.org/abzu/ 


1 


0.0% 


1 


0.1% 


ACADIA (Association for Computer 
Aided Design in Architectur 


http: / / www.acadia.org/ 


1 


0.0% 


1 


0.1% 


AFL-CIO 


http: / /www.aflcio.org/ 


1 


0.0% 


1 


0.1% 


AHEAD (Association on Higher 
Education and Disability) 


http: / / www.ahead.org/ 


1 


0.0% 


1 


0.1% 


AIA American Institute of Architects 


http://www.aia.org/ 


1 


0.0% 


1 


0.1% 


ARCHON - National Archives (UK) 


http:/ /www.archon.nationalarc 
hives.gov.uk/archon/ 


1 


0.0% 


1 


0.1% 


ARTbibliographies Modern 


http://www.csa.com/csa/facts 
heets/ artbm.shtml 


1 


0.0% 


1 


0.1% 


About.com 


About.com 


1 


0.0% 


1 


0.1% 


Ad*Acess 


http: / / scriptorium.lib.duke.edu 
/adaccess/ 


1 


0.0% 


1 


0.1% 


Adavanced Academic 




1 


0.0% 


1 


0.1% 


Adbusters 


http://adbusters.org/home/ 


1 


0.0% 


1 


0.1% 


Adcritic 


http://www.adcritic.com/ 


1 


0.0% 


1 


0.1% 


AddAll Book search and price comparison 


http://www.addall.com/ 


1 


0.0% 


1 


0.1% 


Advanced Papyrological Information 
System (APIS) 


http://www.columbia.edU/cu/l 
web/projects/ digital/apis/ 


1 


0.0% 


1 


0.1% 


Africa Research Program 


http:/ /africa. gov. harvard.edu/ 


1 


0.0% 


1 


0.1% 


Allyn & Bacon/ Longman 


http : / / www .ablongman.com / 


1 


0.0% 


1 


0.1% 


AltaVista 


http:/ /www.altavista.com/ 


1 


0.0% 


1 


0.1% 


American Association of Teachers of 
French 


http: //www Jrenchteachers.org 
/ 


1 


0.0% 


1 


0.1% 


American Library Association 


http: / /www.ala.org/ 


1 


0.0% 


1 


0.1% 


American Meteorological Society 


http: / /www.ametsoc.org/ 


1 


0.0% 


1 


0.1% 


American Presidency Project 


http://www.presidency.ucsb.ed 

u/ 


1 


0.0% 


1 


0.1% 


American Studies Web 


http: / / cfdev.georgetown.edu/c 
ndls/asw/ 


1 


0.0% 


1 


0.1% 


Anarchy Archives 


http:/ / dwardmac.pitzer.edu/ A 
narchist_Archives/ 


1 


0.0% 


1 


0.1% 


AnthroPhoto 


http://www.anthrophoto.com/ 

cgi- 

bin/ImageFolio31/imageFolio.c 


1 


0.0% 


1 


0.1% 


Anthropology in the News 


http:/ /www.tamu.edu/anthrop 
ology/news.html 


1 


0.0% 


1 


0.1% 


Arcat (architecture) 


http: / /www.arcat.com/ 


1 


0.0% 


1 


0.1% 
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Architecture Record 


http:/ / archrecord.construction.c 
om/ 


i 


0.0% 


1 


0.1% 


Archive of Turkish Oral Narrative 


http: //aton. ttu.edu/ 


i 


0.0% 


1 


0.1% 


Archives Network of Alberta Database 


http: / / asalive.archivesalberta.or 
g:8080/access / asa / archaa / 


i 


0.0% 


1 


0.1% 


Archives and Museum Informatics 


www.archimuse.com 


i 


0.0% 


1 


0.1% 


Archives of American Art 


http://archivesofamericanart.si. 

edu/start.htm 


i 


0.0% 


1 


0.1% 


Art Nexus (art magazine) 


http : / / ww w .artnexus. com / inde 
x2.html 


i 


0.0% 


1 


0.1% 


ArtLex Art Dictionary 


http: //artlex.com/ 


i 


0.0% 


1 


0.1% 


ArtUnframed 


http://www.artunframed.com/ 

stuart_2.htm 


i 


0.0% 


1 


0.1% 


ArteHistoria 


http://www.artehistoria.com/ 


i 


0.0% 


1 


0.1% 


Arts and Culture Dictionary, Glossary, 
and Terms Directory 


http://www.glossarist.com/glo 

ssaries/arts-culture/ 


i 


0.0% 


1 


0.1% 


Arts and Humanities Data Service 


http:/ /ahds.ac.uk/ 


i 


0.0% 


1 


0.1% 


Asahi Shinbun (Japanese Newspaper 
online) 


http://www.asahi.com/ 


i 


0.0% 


1 


0.1% 


Associated Press 


http://www.ap.org 


i 


0.0% 


1 


0.1% 


Association for Asian Studies 


http : / / www .aasianst.org/ 


i 


0.0% 


1 


0.1% 


Atlantic Monthly 


http: //www.theatlantic.com 


i 


0.0% 


1 


0.1% 


BeeHive Hypertext/ Hypermedia Literary 
Journal 


http:/ /beehive, temporalimage.c 
om/ 


i 


0.0% 


1 


0.1% 


Berkeley Digital Library Sunsite 


http://sunsite.berkeley.edu/ 


i 


0.0% 


1 


0.1% 


BiblePlaces 


http:/ /www .bibleplaces.com/ 


i 


0.0% 


1 


0.1% 


Biblical Archaeology Society (BAS) 


http://www.basarchive.org/ 


i 


0.0% 


1 


0.1% 


Bibliography of Asian Studies 


http: / / www.aasianst.org/bassu 
b.htm 


i 


0.0% 


1 


0.1% 


Biography Channel 


biography.com 


i 


0.0% 


1 


0.1% 


Board of Governors of the Federal Reserve 
System 


http: //www iederalreserve.gov 
/ 


i 


0.0% 


1 


0.1% 


Britannia 


http: / /www .britannia.com/ 


i 


0.0% 


1 


0.1% 


British Film Institute 


http://www.bfi.org.uk/ 


i 


0.0% 


1 


0.1% 


Brown v. Board of Education Digital 
Archive (ASALH) 


http:/ /kpearson. project.tcnj.edu 
/category/ 


i 


0.0% 


1 


0.1% 


BuddhaNet 


http:/ /buddhanet.net/ 


i 


0.0% 


1 


0.1% 


Bureau of Economic Analysis 


http: / /www .bea.doc.gov/ 


i 


0.0% 


1 


0.1% 


Bureau of Justice Statistics 


http://www.ojp.usdoj.gov/bjs/ 


i 


0.0% 


1 


0.1% 


Bureau of Labor Statistics 


http: / /www.bls.gov/ 


i 


0.0% 


1 


0.1% 


CAADIA (Computer Aided Architectural 
Design in Asia) 


http:/ /www.caadria.org/caadri 

a/ 


i 


0.0% 


1 


0.1% 


CERES California Environmental 
Resources Evaluation System 


http:/ / ceres.ca.gov/ 


i 


0.0% 


1 


0.1% 


CLOE - Co-operative Learning Object 
Exchange 


http:/ / cloe.on.ca/ 


i 


0.0% 


1 


0.1% 


CPJ Freedom Press Online 


http://www.cpj.org/ 


i 


0.0% 


1 


0.1% 


CROW Course Resources on the Web 


http://www.millikin.edu/aci/c 

row/ 


i 


0.0% 


1 


0.1% 


CUMIN CAD (Cumulative Index of 
Computer Aided Architectural D 


http: / / cumincad.scix.net/ 


i 


0.0% 


1 


0.1% 


California Association for Postsecondary 
Education and Disab 


http: / / www.caped.net/ 


i 


0.0% 


1 


0.1% 


California Courts: Rules 


http:/ / www .courtinfo .ca. gov / r 
ules/ 


i 


0.0% 


1 


0.1% 


California Ethnic and Multicultural 
Archives 


http: / / cemaweb.library .ucsb.ed 
u/ cema_index.html 


i 


0.0% 


1 


0.1% 
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Camoes Institute website 


http:/ /www.instituto- 
camoes.pt/ 


i 


0.0% 


1 


0.1% 


Carol Gerten's Fine Art CGFA: A virtual 
art museum 


http:/ / cgfa.sunsite.dk/ 


i 


0.0% 


1 


0.1% 


Celtic Studies Association of North 
America on-line bibliogr 


http://www.humnet.ucla.edu/ 

humnet/celtic/csanabib.html 


i 


0.0% 


1 


0.1% 


Center for American Women and Politics 


http://www.cawp.mtgers.edu/ 


i 


0.0% 


1 


0.1% 


Center for Budgets and Policy Priorities 


http: //www.cbpp.org/ 


i 


0.0% 


1 


0.1% 


Center for Land Use Information 


http://clui.org/ 


i 


0.0% 


1 


0.1% 


Center for Technology in Government 


http: //www.ctg.albany.edu/ 


i 


0.0% 


1 


0.1% 


Center for the Study of Ancient 
Documents 


http: / / www.csad.ox.ac.uk/ CSA 

D/ 


i 


0.0% 


1 


0.1% 


Center for the Study of the American 
South 


http: / / www.unc .edu/ depts /csa 

s/ 


i 


0.0% 


1 


0.1% 


Child Trends 


http: //www.childtrends.org/ 


i 


0.0% 


1 


0.1% 


Chimpanzoo 


http: //www.chimpanzoo.org/ 


i 


0.0% 


1 


0.1% 


Chinese and Japanese Art History WWW 
Virtual Library 


http: / / www.nyu.edu/gsas/ dep 
t / f ineart/ html / Chinese/ index . ht 
ml 


i 


0.0% 


1 


0.1% 


Christian Classics Ethereal Library 


http: //www.ccel.org/ 


i 


0.0% 


1 


0.1% 


Christian Science Monitor 


http: //csmonitor.com 


i 


0.0% 


1 


0.1% 


Civil War Photos 


http://www.civilwarphotos.net 
/ files/group_photos.htm 


i 


0.0% 


1 


0.1% 


Classroom.com 


classroom.com 


i 


0.0% 


1 


0.1% 


Cleansurface 


cleansurface.org "living archive 
of public trouble-making, ..." 


i 


0.0% 


1 


0.1% 


Columbia Encyclopedia Online 


available through yahoo or 
bartleby 


i 


0.0% 


1 


0.1% 


Common-place 


http: / /common-place. org/ 


i 


0.0% 


1 


0.1% 


Computer Aided Instruction Program 


http:/ /cai.ucdavis.edu/ 


i 


0.0% 


1 


0.1% 


Congressional Biography 


http: / /bioguide. congress. gov 


i 


0.0% 


1 


0.1% 


Conjuguemos 


conjuguemos.com 


i 


0.0% 


1 


0.1% 


Connecticut History Online 


http://www.cthistoryonline.org 

/ 


i 


0.0% 


1 


0.1% 


Coolhomepages (webdesign) 


http:/ /www.coolhomepages.co 
m/ 


i 


0.0% 


1 


0.1% 


Corpus Christianorum 


http:/ /www.corpuschristianoru 
m.org/home.html 


i 


0.0% 


1 


0.1% 


Cultura 


http://web.mit.edu/french/cult 

uraNEH/ 


i 


0.0% 


1 


0.1% 


Cumulus Digital Image Database 




i 


0.0% 


1 


0.1% 


Cuneiform Digital Library Initiative 


http: / /cdli.ucla.edu/ 


i 


0.0% 


1 


0.1% 


CyberJournalist.net 


http: / /cyberjournalist.net/ 


i 


0.0% 


1 


0.1% 


Cyberspace and Critical Theory 


http:/ /www.cyberartsweb.org/ 
cpace/ 


i 


0.0% 


1 


0.1% 


DASL Data and Story Library 


http://lib.stat.cmu.edu/DASL/ 


i 


0.0% 


1 


0.1% 


DV (Digital Video) 


http://dv.com/ 


i 


0.0% 


1 


0.1% 


Daily Dutch News broadcast archive 


http: / /www .nos.nl/ 


i 


0.0% 


1 


0.1% 


David Rumsey Historical Map Collection 


http:/ /www. davidrumsey.com 


i 


0.0% 


1 


0.1% 


De imperatoribus romanis (online 
Encyclopedia of Roman Emper 


http :/ / www .roman- 
emperors.org/ 


i 


0.0% 


1 


0.1% 


Dictionary.com 


Dictionary.com 


i 


0.0% 


1 


0.1% 


DigiZeitschriften 


http: / / www.digizeitschriften.de 
/ 


i 


0.0% 


1 


0.1% 


Digital History 


http:/ /www.digitalhistory.uh.e 
du/ 


i 


0.0% 


1 


0.1% 
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Digital National Security Archive 
(Proquest) 


http: / /nsarchive.chadwyck.com 
/ 


i 


0.0% 


1 


0.1% 


Digitale Bibliothek Deutscher Klassiker 


http: //klassiker.chadwyck.com 
/ 


i 


0.0% 


1 


0.1% 


Diotima: Women and Gender in the 
Ancient World 


http://www.stoa.org/ diotima/ 


i 


0.0% 


1 


0.1% 


Discovery Channel 


discovery.com 


i 


0.0% 


1 


0.1% 


Dogpile (search engine page) 


http: / / www. dogpile.com/ 


i 


0.0% 


1 


0.1% 


EBSCOHost 


http: / / ejournals.ebsco.com/disc 
laimer.asp 


i 


0.0% 


1 


0.1% 


EH.Net- Economic History Services 


http://eh.net/ 


i 


0.0% 


1 


0.1% 


ESRI - GIS Software 


http: / /esri.com/ 


i 


0.0% 


1 


0.1% 


ETOH database 


http://etoh.niaaa.nih.gov/ 


i 


0.0% 


1 


0.1% 


Early Modem England Sources 


http : / / www .quelle . or g/ ernes / r 
esearch.html 


i 


0.0% 


1 


0.1% 


Ecole Initiative 


http: / / www2.evansville.edu/ ec 
oleweb/ 


i 


0.0% 


1 


0.1% 


EconLit 


http://www.econlit.org/ 


i 


0.0% 


1 


0.1% 


Economic History Services 


http://eh.net/ 


i 


0.0% 


1 


0.1% 


Edsource (California) 


http:/ /www.edsource.org/ 


i 


0.0% 


1 


0.1% 


Education Data Partnership (California) 


http://www.ed- 

data.kl2.ca.us/welcome.asp 


i 


0.0% 


1 


0.1% 


El Genero en Historia/ Gender in History 


http: / / www.sas.ac.uk/ ilas/ gen 
ero.htm 


i 


0.0% 


1 


0.1% 


Electric Library 


http:/ /www.electriclibrary.com 
/ 


i 


0.0% 


1 


0.1% 


Electronic Literature Organization 
(eliterature) 


http:// www.eliterature.org/ 


i 


0.0% 


1 


0.1% 


Electronic Text Corpus of Sumerian 
Literature 


http:/ /www- 
etcsl.orient.ox.ac.uk/ 


i 


0.0% 


1 


0.1% 


Encyclopedia.com 


http: // www.encyclopedia.com/ 


i 


0.0% 


1 


0.1% 


Environmental Protection Agency 


http://epa.gov/ 


i 


0.0% 


1 


0.1% 


Equal Employment Opportunity 
Commission 


http://www.eeoc.gov/ 


i 


0.0% 


1 


0.1% 


Etcetera (online Mexican magazine) 


http://etcetera.com.mx / 


i 


0.0% 


1 


0.1% 


Eugenics Archive 


http://www.eugenicsarchive.or 

g/ 


i 


0.0% 


1 


0.1% 


Eurasianet 


http://www.eurasianet.org/ 


i 


0.0% 


1 


0.1% 


EuroDocs: Western European Primary 
Historical Documents 


http:/ /library .byu.edu/ ~rdh/ e 
urodocs/ 


i 


0.0% 


1 


0.1% 


European Journalism Center 


http://www.ejc.nl/ 


i 


0.0% 


1 


0.1% 


European Social Survey 


http:// www.europeansocialsur 
vey.org/ 


i 


0.0% 


1 


0.1% 


FAMSF Fine Arts Museum of San 
Francisco 


http: / / www.thinker.org/ index, 
asp 


i 


0.0% 


1 


0.1% 


Fabula 


http: / /www iabula.org/ 


i 


0.0% 


1 


0.1% 


Factiva 


http: / /www iactiva.com/ 


i 


0.0% 


1 


0.1% 


Fannie Mae 


http : / / www .fanniemae .com/ in 
dex.jhtml 


i 


0.0% 


1 


0.1% 


Fayum Project 


http: / / fayum.arts.kuleuven.ac.b 

e/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Atlanta 


http: / /www irbatlanta.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Boston 


http: //www .bos.frb.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Chicago 


http://www.chicagofed.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Cleveland 


http: / / www.clevelandfed.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Dallas 


http: //www. dallasfed.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Kansas City 


http: //www.kansascityfed.org/ 


i 


0.0% 


1 


0.1% 
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Federal Reserve Bank - Minneapolis 


http:/ /www.minneapolisfed.or 

g / 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - New York 


http: / /www .newyorkfed.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Philadelphia 


http:/ /www.philadelphiafed.or 

g / 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - Richmond 


http://www.rich.frb.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - San Francisco 


http:/ /www irbsf.org/ 


i 


0.0% 


1 


0.1% 


Federal Reserve Bank - St. Louis 


http:/ /www.stlouisfed.org/ 


i 


0.0% 


1 


0.1% 


Financial Times 


http: / /news.ft.com/home/us 


i 


0.0% 


1 


0.1% 


Finfo (Finnish Information Site) 


http : / / www .finfo . dk/ 


i 


0.0% 


1 


0.1% 


First Monday (online journal) 


http:/ /www iirstmonday.dk/ 


i 


0.0% 


1 


0.1% 


Flash Kit, a flash developer resource 


http: / / www.flashkit.com/ index 
.shtml 


i 


0.0% 


1 


0.1% 


Foto Marburg 


http://www.fotomarburg.de/in 

dex_e.htm 


i 


0.0% 


1 


0.1% 


Foundation for the Advancement of 
Mesoamerican Studies, Inc. 


http : / / www .famsi . or g/ 


i 


0.0% 


1 


0.1% 


Fox 


http: / /www ioxnews.com/ 


i 


0.0% 


1 


0.1% 


France's Institut National de 
l'Audiovisuel 


http : / / www .ina .fr 


i 


0.0% 


1 


0.1% 


French Colonial Archives 




i 


0.0% 


1 


0.1% 


French in Action resource website 


http: / / yalepress.yale.edu/yupb 
ooks / Fi A/ FrenchinAction.html 


i 


0.0% 


1 


0.1% 


GGRENir - Intemetography on 
Renaissance Intellectual Histor 


http : / / www .phil-hum-ren. uni- 
muenchen.de/GGRENirDB / 


i 


0.0% 


1 


0.1% 


Galileo Project 


http: / / galileo.rice.edu/ 


i 


0.0% 


1 


0.1% 


Gallica 


http://gallica.bnf.fr/ 


i 


0.0% 


1 


0.1% 


General Social Survey (GSS) by National 
Opinion Research Cen 


http:// www.norc.uchicago.edu 
/ projects / gensoc .asp 


i 


0.0% 


1 


0.1% 


GeoCritica 


http://www.ub.es/ geocrit/men 
u.htm 


i 


0.0% 


1 


0.1% 


Geography Departments Worldwide 


http://univ.cc/geolinks/ 


i 


0.0% 


1 


0.1% 


George Eastman House 


http:/ /www.eastmanhouse.org 
/ 


i 


0.0% 


1 


0.1% 


German Propaganda Archive 


http: / / www.calvin.edu/ acade 
mic/cas/gpa/ 


i 


0.0% 


1 


0.1% 


Getty Images 


http: //www. gettyimages.com 


i 


0.0% 


1 


0.1% 


Gilder Lehrman Institute of American 
History 


http://www.gilderlehrman.org 

/ 


i 


0.0% 


1 


0.1% 


Glenbow Museum 


http: / / www.glenbow.org/ 


i 


0.0% 


1 


0.1% 


Glowlab 


http: / / glowlab.blogs.com/ 


i 


0.0% 


1 


0.1% 


Goethe Institute 


http: / / www.goethe.de/ enindex 
.htm 


i 


0.0% 


1 


0.1% 


Goo (online Japanese dictionary) 


http://dictionary.goo.ne.jp/ 


i 


0.0% 


1 


0.1% 


Government Printing Office 


http:// www.gpoaccess.gov/ind 
ex.html 


i 


0.0% 


1 


0.1% 


Government Resources 


http:/ /www.lib.umich.edu/gov 
docs/ 


i 


0.0% 


1 


0.1% 


Greek Grammar 


http: //perswww. kuleuven.ac.be 
/ ~u0013314/ greekg.htm 


i 


0.0% 


1 


0.1% 


Grizzard: Construction of UVA 


http:/ / etext.virginia.edu/jeffers 
on/grizzard/ 


i 


0.0% 


1 


0.1% 


Hanover Historical Texts Project 


http://history.hanover.edu/proj 

ect.html 


i 


0.0% 


1 


0.1% 


HarpWeek 


http: //www.harpweek.com/ 


i 


0.0% 


1 


0.1% 


Harry Ransom Humanities Research 
Center (HRC) 


http:// www.hrc.utexas.edu/ 


i 


0.0% 


1 


0.1% 
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Hartman Center for Sales, Advertising 
and Marketing History 


http: / / scriptorium.lib.duke.edu 
/hartman/ 


i 


0.0% 


1 


0.1% 


Harvey Goldberg Center 


http://history.wisc.edu/Goldbe 
rg/ Legacy.htm 


i 


0.0% 


1 


0.1% 


Heidelberger Gesamtverzeichnis 


http : / / www .rzuser .uni- 
heidelberg. de/ ~gvO / gvz.html 


i 


0.0% 


1 


0.1% 


Historical Text Archive 


http: / /historicaltextarchive.com 
/ 


i 


0.0% 


1 


0.1% 


History News Network 


http://hnn.us/ 


i 


0.0% 


1 


0.1% 


History of Science Society 


http: / /www .hssonline.org/ 


i 


0.0% 


1 


0.1% 


Hodge's Health Center - Care Domains - 
Model 


http://www.p- 

jones.demon.co.uk/ 


i 


0.0% 


1 


0.1% 


Hoover's Online: The Business 
Information Authority 


hoovers.com 


i 


0.0% 


1 


0.1% 


Humbul Humanities Hub 


http : / / www .humbul. ac .uk / 


i 


0.0% 


1 


0.1% 


Huntington Archive (Buddhist art) 


http: //kaladarshan.arts.ohio- 
state.edu/ 


i 


0.0% 


1 


0.1% 


Hypertext Preprocessor 


http: / /www.php.net/ 


i 


0.0% 


1 


0.1% 


|IESBS (International Encyclopedia of the Social and Behavior 


i 


0.0% 


1 


0.1% 


IMF International Monetary Fund 


http: / /www.imf .org/ 


i 


0.0% 


1 


0.1% 


IPU Inter-Parliamentary Union 


http:/ / www.ipu.org/english/ h 
ome.htm 


i 


0.0% 


1 


0.1% 


1ST AT? Integrating Science Teaching and 
Technology 


http:// www.ucmp.berkeley.edu 
/IU/ 


i 


0.0% 


1 


0.1% 


Indiana Digital Images Collection 




i 


0.0% 


1 


0.1% 


Indiana University Languages Online 


http://languagelab.bh.indiana.e 

du/korean.html 


i 


0.0% 


1 


0.1% 


Information Wissenschaft & Praxis 


http://www.dgd.de/dgi/nfd/ 


i 


0.0% 


1 


0.1% 


Inner City Struggle 


schoolsnotjails.com 


i 


0.0% 


1 


0.1% 


Interactive Web Casting 




i 


0.0% 


1 


0.1% 


| International Bibliography of the Social Sciences (IBSS data 


i 


0.0% 


1 


0.1% 


International Public Management 
Association for Human Resour 


http:/ /www.ipma-hr.org/ 


i 


0.0% 


1 


0.1% 


Internet Movie Database 


http: //www.imdb.com/ 


i 


0.0% 


1 


0.1% 


Internet Scout Project 


http: / / scout, wise .edu/ 


i 


0.0% 


1 


0.1% 


Islamicity - Islam and the Global Muslim 
eCommunity 


http://www.islamicity.com/ 


i 


0.0% 


1 


0.1% 


Iter: Gateway to the Middle Ages and 
Rennaisance 


http: / / www.itergateway.org/ 


i 


0.0% 


1 


0.1% 


Jack Lynch's Guide to Grammar and Style 


http:// andromeda.rutgers.edu/ 
-jlynch/ Writing/ 


i 


0.0% 


1 


0.1% 


Jane Goodall Institute 


http://www.janegoodall.org/ 


i 


0.0% 


1 


0.1% 


Japanese Text Initiative 


http:/ / etext.lib.virginia.edu/jap 
anese/ 


i 


0.0% 


1 


0.1% 


Jensen's Scholar's Guide to Humanities 
and Social Sciences 


http:// tigger.uic.edu/~rjensen/ 


i 


0.0% 


1 


0.1% 


Johannine Literature (Felix Just) 


http:// myweb.lmu.edu/fjust/Jo 
hn.htm 


i 


0.0% 


1 


0.1% 


Kamusi Project (Swahili Dictionary) 


www.yale.edu/swahili/home.ht 

ml 


i 


0.0% 


1 


0.1% 


Kenkyusha (online Japanese dictionary) 


http:/ /www.kenkyusha.co.jp/o 
nline-dic /on-dicl.html 


i 


0.0% 


1 


0.1% 


Kinsey Institute Library 


http:/ / www.indiana.edu/~kins 
ey / library / library.html 


i 


0.0% 


1 


0.1% 


Korean History: A Bibliography 


http:/ /www2.hawaii.edu/korea 
/ biblio graphy / biblio .htm 


i 


0.0% 


1 


0.1% 


Korean Studies at Sogang 


http://korean.sogang.ac.kr/ 


i 


0.0% 


1 


0.1% 
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Korean daily newspaper site 1 


http: / / donga.com/ 


i 


0.0% 


1 


0.1% 


Korean daily newspaper site 2 


http:/ /www.chosun.com/ 


i 


0.0% 


1 


0.1% 


L'Annee philologique 


http:/ /www.annee- 
philologique.com/aph/ 


i 


0.0% 


1 


0.1% 


LDA Learning Disabilities Association 


http://www.ldanatl.org/ 


i 


0.0% 


1 


0.1% 


LEO German English Dictionary 


http: / / dict.leo.org/ 


i 


0.0% 


1 


0.1% 


LVR Mai-Tagung 


www.mai-tagung.de 


i 


0.0% 


1 


0.1% 


Le Monde (daily news) 


http: //www.lemonde.fr 


i 


0.0% 


1 


0.1% 


Le Monde diplomatique (monthly 
newspaper) 


http:/ /www.monde- 
diplomatique.fr 


i 


0.0% 


1 


0.1% 


Learn Spanish 


http://www.studyspanish.com 

/ 


i 


0.0% 


1 


0.1% 


Legal Information Institute (LII) 


http: / /www.law. cornell.edu/ 


i 


0.0% 


1 


0.1% 


Lenin Internet Archive 


http://www.marxists.org/archi 

ve/lenin/ 


i 


0.0% 


1 


0.1% 


Leuven Database of Ancient Books 


http://ldab.arts.kuleuven.ac.be/ 


i 


0.0% 


1 


0.1% 


Leuven Database of Papyrus Collections 
Worldwide 


http :// lhpc .arts . kuleuven. ac .be/ 
collections/ 


i 


0.0% 


1 


0.1% 


Lexikon des Mittelalters 


http:/ /www.uni- 

tuebingen. de/ ub/ db/ index .htm 

?http:/ /silvanus. 


i 


0.0% 


1 


0.1% 


Liberature, newspaper 


http: / /www.liberation.fr 


i 


0.0% 


1 


0.1% 


Lindesmith 


http://library.soros.org/lindes 

mith.html 


i 


0.0% 


1 


0.1% 


Living Landscapes 


http: / / www.livinglandscapes.b 
c.ca/ 


i 


0.0% 


1 


0.1% 


Ljud och Bild (Sound and Image) Swedish 
government archive 


http: / / www.ljudochbildarkivet. 
se 


i 


0.0% 


1 


0.1% 


Lycos 


http://www.lycos.com/ 


i 


0.0% 


1 


0.1% 


MACTiA 


http://www.mactia.berkeley.ed 

u/ 


i 


0.0% 


1 


0.1% 


MIT Open Course Ware 


http:/ /ocw.mit.edu/index.html 


i 


0.0% 


1 


0.1% 


Macromedia 


macromedia.com 


i 


0.0% 


1 


0.1% 


Mannheim University Social Science Data 
Archive 


http:/ / www.mzes.uni- 

mannheim.de/frame.php?oben= 

titel_e.html& 


i 


0.0% 


1 


0.1% 


Maplnfo 


http://www.mapinfo.com/ 


i 


0.0% 


1 


0.1% 


Mapquest 


mapquest.com 


i 


0.0% 


1 


0.1% 


Maryland Faculty Online 


http://www.mdfaconline.org/ 


i 


0.0% 


1 


0.1% 


Masters of Photography 


http:/ /www.masters-of- 
photography.com/ 


i 


0.0% 


1 


0.1% 


McConnell/The Good Earth 


http :/ / www .rnhhe .com / earthsc 
i/ geology/mcconnell/ 


i 


0.0% 


1 


0.1% 


Medical Humanities Resource Database 


http://www.mhrd.ucl.ac.uk/ 


i 


0.0% 


1 


0.1% 


Medline database 


http:/ /medline.cos.com/ 


i 


0.0% 


1 


0.1% 


Mesolore 


http:/ /www.mesolore.com/ho 
me. htm 


i 


0.0% 


1 


0.1% 


Microscopy Society of America 


http://www.msa.microscopy.co 

m/ 


i 


0.0% 


1 


0.1% 


Middle East Maps (Dartmouth) 


http:// www.dartmouth.edu/~g 
ov46/ 


i 


0.0% 


1 


0.1% 


Ministere de la Culture 


http://www.culture.gouv.fr/ 


i 


0.0% 


1 


0.1% 


Monash University Arts 


http: / / www.arts.monash.edu.a 
u/korean/ 


i 


0.0% 


1 


0.1% 


Mudcat Cafe 


http:// www.mudcat.org/alltitle 
s.cfm 


i 


0.0% 


1 


0.1% 
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Musee Nicephore Niepce 


http:/ /www.museeniepce.com/ 
indexplorer.html 


i 


0.0% 


1 


0.1% 


Museum national d'Histoire naturelle 
(Musee de 1'Homme Paris 


http://www.mnhn.fr/ 


i 


0.0% 


1 


0.1% 


Museum of Modem Art 


http: //www.moma.org/ 


i 


0.0% 


1 


0.1% 


NCSC - National Center for State Courts 


http: / /www .ncsconline.org/ 


i 


0.0% 


1 


0.1% 


NGDC National Geophysical Data Center 
GLOBE 


http: / / www.ngdc.noaa.gov/ seg 
/ topo / globe . shtml 


i 


0.0% 


1 


0.1% 


NKCA Neighborhood Knowledge 
California 


http:// www.nkca.ucla.edu/ 


i 


0.0% 


1 


0.1% 


NO A A National Weather Service 


http:/ /www .nws.noaa.gov/ 


i 


0.0% 


1 


0.1% 


Nanzan Institute for Religion and Culture 


http :/ / www .ic .nanzan- 
u.ac.jp/ SHUBUNKEN / 


i 


0.0% 


1 


0.1% 


National Atlas of the USA 


http : / / www .nationalatlas . gov/ 


i 


0.0% 


1 


0.1% 


National Bureau of Economic Research 
(NBER) 


http:// www.nber.org/ 


i 


0.0% 


1 


0.1% 


National Center for Digital Government 
(Harvard) 


http://www.ksg.harvard.edu/ d 
igitalcenter/ 


i 


0.0% 


1 


0.1% 


National Data Buoy Center 


http: / /www .ndbc.noaa.gov / 


i 


0.0% 


1 


0.1% 


National Gallery of Art 


http:/ /www. nga.gov/ 


i 


0.0% 


1 


0.1% 


National Gallery of Australia (NGA) 


http : / / www .nga . gov .au/ Home 
/index.cfm 


i 


0.0% 


1 


0.1% 


National Geographic Online 


http:// www.nationalgeographic 
.com/ 


i 


0.0% 


1 


0.1% 


National Library of Scotland 


http : / / www .nls.uk/ 


i 


0.0% 


1 


0.1% 


National Portrait Gallery 


http://www.npg.org.uk/live/in 

dex.asp 


i 


0.0% 


1 


0.1% 


Nettime mailing lists 


http:/ /www .nettime.org/ 


i 


0.0% 


1 


0.1% 


New Advent Catholic Encyclopedia 
Online 


http://www.newadvent.org/cat 

hen/ 


i 


0.0% 


1 


0.1% 


New Deal Library 


http: //newdeal.feri.org/index.h 
tm 


i 


0.0% 


1 


0.1% 


New York Public Library's Picture 
Collection Online 


http:// digital.nypl.org/mmpco 
/ 


i 


0.0% 


1 


0.1% 


New York Review of Books 


http://www.nybooks.com/ 


i 


0.0% 


1 


0.1% 


Newsbank 


http: / /www .newsbank.com/ 


i 


0.0% 


1 


0.1% 


Nuts and Bolts of College Writing 


http: / / nutsandbolts.washcoll.ed 

u/ 


i 


0.0% 


1 


0.1% 


Office of Personnel Management 


http: / /www.opm.gov/ 


i 


0.0% 


1 


0.1% 


OhioLINK 


http: / /www.ohiolink.edu/ 


i 


0.0% 


1 


0.1% 


Online Medelian Inheritance in Man 
(OMIM) 


http: / / www.ncbi.nlm.nih.gov/e 
ntrez / query .fcgi? db=OMIM 


i 


0.0% 


1 


0.1% 


Ontario Archives 


http://www.archives.gov.on.ca 

/ 


i 


0.0% 


1 


0.1% 


Our Future, Our Past 


http:/ /www.ourfutureourpast.c 

a/ 


i 


0.0% 


1 


0.1% 


OurRoots/Nos Racines: The Canadian 
Digital Local Histories S 


http: / / www.ourroots.ca/ 


i 


0.0% 


1 


0.1% 


Oxford Dictionary of National Biography 


http://www.oup .com / oxfordd 
nb/info/ 


i 


0.0% 


1 


0.1% 


Oxford Digital Library 


http://www.odl.ox.ac.uk/ 


i 


0.0% 


1 


0.1% 


Oxford Text Archive 


http://ota.ahds.ac.uk/ 


i 


0.0% 


1 


0.1% 


PAL: Perspectives in American Literature 


http://www.csustan.edu/englis 
h / reuben / pal/ TABLE.HTML 


i 


0.0% 


1 


0.1% 


PASSIA (Palestinian Academic Society for 
the Study of Intern 


www.passia.org 


i 


0.0% 


1 


0.1% 


Patriagrande 


http://www.patriagrande.net/ 


i 


0.0% 


1 


0.1% 
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Peabody Museum 


http: / / www.peabody .harvard, e 
du/ 


i 


0.0% 


1 


0.1% 


Pennsylvania Sumerian Dictionary 


http:/ /psd. museum.upenn.edu 
/ 


i 


0.0% 


1 


0.1% 


Phronesis: Ancient and Modern 
Philosophy 


http:// phronesis.org/ 


i 


0.0% 


1 


0.1% 


PhysicalGeography 


http:// www.physicalgeography 
.net/ 


i 


0.0% 


1 


0.1% 


PiCarta 


http://picarta.pica.nl/ 


i 


0.0% 


1 


0.1% 


Picture History 


http: / / www.picturehistory.com 
/ 


i 


0.0% 


1 


0.1% 


Poole's Index to Periodical Literature 




i 


0.0% 


1 


0.1% 


Postcolonial and Postimperial Literature 


http: //www.postcolonialweb.or 

g / 


i 


0.0% 


1 


0.1% 


Precolumbian Art Research Institute 


http://www.mesoweb.com/par 

i/ 


i 


0.0% 


1 


0.1% 


Premodern Japanese Studies (PMJS) 
archive and list 


http:// www.meijigakuin.ac.jp/ 
-prnjs/ 


i 


0.0% 


1 


0.1% 


Prentice Hall Publisher 


http: / /vig.prenhall.com/ 


i 


0.0% 


1 


0.1% 


Prentice Hall Publisher's companion 
website 


http://www.prenhall.com/pow 

ell/ 


i 


0.0% 


1 


0.1% 


Primate Gallery 


http:/ / staff.washington.edu/ ti 
mk / primate / main.html 


i 


0.0% 


1 


0.1% 


Princeton Dante Project 


http:/ / etcweb.princeton.edu/ da 
nte/ index.html 


i 


0.0% 


1 


0.1% 


PubMed (two possible websites both are 
NIH related) 


http:/ /www.pubmedcentral.nih 
•gov/ 


i 


0.0% 


1 


0.1% 


Questia 


http://www.questia.com/ 


i 


0.0% 


1 


0.1% 


RFE/RL (Radio Free Europe/Radio 
Liberty) 


http: / / www.rferl.org/ 


i 


0.0% 


1 


0.1% 


RLG Cultural Materials 


http://www.rlg.org/ 


i 


0.0% 


1 


0.1% 


RaceSci: History of Race in Science 


http: / /www.racesci.org/ 


i 


0.0% 


1 


0.1% 


Radio France internationale (radio 
website) 


http: //www.rfi.fr 


i 


0.0% 


1 


0.1% 


Refworks 


http: / /refworks.com/ 


i 


0.0% 


1 


0.1% 


Religious Liberty Archive 


http://www.churchstatelaw.co 

m/ 


i 


0.0% 


1 


0.1% 


Resources for Earth Science and 
Geography Instruction 


http: / / webs.cmich.edu/resgi/ 


i 


0.0% 


1 


0.1% 


Roper Center for Public Opinion Research 


http://www.ropercenter.uconn. 

edu/ 


i 


0.0% 


1 


0.1% 


Runme 


http: / / runme.org/ 


i 


0.0% 


1 


0.1% 


Rutgers 


http: / /www.rutgers.edu/ 


i 


0.0% 


1 


0.1% 


SAH (Society of Architectural Historians): 
Image Exchange 


http://www.sah.org/index.php 

?module=ContentExpress&func 

=disp 


i 


0.0% 


1 


0.1% 


SF Museum of Modem Art 


http: //www.sfmoma.org/ 


i 


0.0% 


1 


0.1% 


SIRS School Improvement Research Series 


http://www.nwrel.org/scpd/si 

rs/ 


i 


0.0% 


1 


0.1% 


Salon.com 


salon.com 


i 


0.0% 


1 


0.1% 


San Jose State's Image Repository 


http://worldart.sjsu.edu/ 


i 


0.0% 


1 


0.1% 


Saskia Ltd 


http:/ /www.saskia.com 


i 


0.0% 


1 


0.1% 


ScienceDirect 


http: / /www.sciencedirect.com/ 


i 


0.0% 


1 


0.1% 


Senate Historical Office 


http:// www.senate.gov/artand 
history / history /common / gener 
ic/S 


i 


0.0% 


1 


0.1% 
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Shakespeare Sources 


http:// www.shakespeare- 
w .com/ english/ Shakespeare / so 
urce.html 


i 


0.0% 


1 


0.1% 


Shifting Baselines 


http: // www.shiftingbaselines.or 
g/index.php 


i 


0.0% 


1 


0.1% 


Slavery Images - University of Virginia 
(UVA) 


http: //hitchcock.itc.vir ginia.edu 
/Slavery/ 


i 


0.0% 


1 


0.1% 


Social Science Research Network (SSRN) 


http:/ /www.ssracom/ 


i 


0.0% 


1 


0.1% 


Societe Francaise de Photographie 


http: / / www.sfp.photographie.c 
om/ 


i 


0.0% 


1 


0.1% 


Society for Medical Anthropology 


http : / / medanthro .net/ 


i 


0.0% 


1 


0.1% 


Sor Juana Inez de la Cruz Project 


http:/ /www. dartmouth.edu/~s 
orjuana/ 


i 


0.0% 


1 


0.1% 


Southern California Swell Model 


http://cdip.ucsd.edu/ 


i 


0.0% 


1 


0.1% 


Stanford Encyclopedia of Philosophy 


http: / / plato.stanford.edu 


i 


0.0% 


1 


0.1% 


Stanford University Jguide 


http://jguide.stanford.edu/ 


i 


0.0% 


1 


0.1% 


State Library - South Australiana 
(Mortlock Library) 


http: / / www.catalog.slsa.sa.gov. 
au:1084/screens/opacmenu.htm 
1 


i 


0.0% 


1 


0.1% 


Stateline.org 


http: / / www.stateline.org/statel 
ine/ 


i 


0.0% 


1 


0.1% 


Storytellers: Native American Authors 
Online 


http: / /www .hanksville.org/stor 
ytellers/ 


i 


0.0% 


1 


0.1% 


Strindberg Museum 


http://www.strindbergsmuseet. 
se/ index_eng . html 


i 


0.0% 


1 


0.1% 


TBRC - Tibetan Buddhist Resource Center 


http: / /www.tbrc.org/ 


i 


0.0% 


1 


0.1% 


TEAMS Middle English Text 


http://www.teamsmedieval.org 

/texts/ 


i 


0.0% 


1 


0.1% 


Talk Origins 


http: / /www.talkorigins.org/ 


i 


0.0% 


1 


0.1% 


Tate Museum 


http:// www.tate.org.uk/home/ 
default.htm 


i 


0.0% 


1 


0.1% 


Teaching Tolerance 


http: / /www.tolerance.org/ 


i 


0.0% 


1 


0.1% 


Tekom 


www.tekom.de 


i 


0.0% 


1 


0.1% 


Tennessee Bob's Famous French Links 


http:// www.utm.edu/departm 
ents/french/french.html 


i 


0.0% 


1 


0.1% 


TerraServer 


http: / /terraserver-usa.com/ 


i 


0.0% 


1 


0.1% 


The Black Elk Speaks 


http://www.blackelkspeaks.unl 

.edu/ 


i 


0.0% 


1 


0.1% 


The British Museum 


http :/ / www . thebritishmuseum . 
ac.uk/ 


i 


0.0% 


1 


0.1% 


The Brookings Institution 


http: / /www .brookings.edu/ 


i 


0.0% 


1 


0.1% 


The Century Foundation 


http: //www.tcf.org/ 


i 


0.0% 


1 


0.1% 


The Daguerreotype: the Daugerreian 
Society 


http:// www.daguerre.org/ 


i 


0.0% 


1 


0.1% 


The Decameron Web 


http://www.brown.edu / Depart 
ments / Italian_Studies / dweb / d 
web.s 


i 


0.0% 


1 


0.1% 


The Economist 


http:/ /www.economist.com/ 


i 


0.0% 


1 


0.1% 


The Forest History Society Databases 


http://www.lib.duke.edu/fores 
t / Research/ databases .html 


i 


0.0% 


1 


0.1% 


The Founders' Constitution 


http: //press- 

pubs. uchicago.edu/founders/ 


i 


0.0% 


1 


0.1% 


The Hermitage Museum 




i 


0.0% 


1 


0.1% 


The Kanji Site 


http: / /www.kanjisite.com/ 


i 


0.0% 


1 


0.1% 


The Medieval Review (TMR) 


http:// www.hti.umich.edu/t/ 1 
mr/ 


i 


0.0% 


1 


0.1% 
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The National Gallery, London 


http:/ /www .nationalgallery.org 
.uk/ 


i 


0.0% 


1 


0.1% 


The Old Bailey Online 


http: / /www.oldbaileyonline.or 
6 / 


i 


0.0% 


1 


0.1% 


The Online Books Page 


http:/ / digital.library.upenn.edu 
/books/ 


i 


0.0% 


1 


0.1% 


The Oriental Institute 


http : / / oi . uchicago . edu/ OI/ def a 
ult.html 


i 


0.0% 


1 


0.1% 


The Philosophy Pages/ Die Philosophie- 
Seiten 


http: / / www.philo.de/Philosop 
hie-Seiten/ 


i 


0.0% 


1 


0.1% 


The Society for Political Methodology 


http://polmeth.wustl.edu/ 


i 


0.0% 


1 


0.1% 


The Teaching Resources Center Online 
Writing Project 


http:/ / cai.ucdavis.edu/trc/trcd 
efault.html 


i 


0.0% 


1 


0.1% 


The Urban Institute 


http://www.urban.org/ 


i 


0.0% 


1 


0.1% 


The World and I 


http://www.worldandi.com/ 


i 


0.0% 


1 


0.1% 


The Writings of Charles Darwin 


http: //pages.britishlibrary.net/ c 
harles.darwin/ 


i 


0.0% 


1 


0.1% 


Thursby's Religious Studies at University 
of Florida 


http://www.clas.ufl.edu/users/ 
gthursby/ rel/ 


i 


0.0% 


1 


0.1% 


Topozone 


http: / /topozone.com/ 


i 


0.0% 


1 


0.1% 


Transcendentalists 


http : / / www .transcendentalists. 
com/ 


i 


0.0% 


1 


0.1% 


Typography.com 


http://typography.com/ 


i 


0.0% 


1 


0.1% 


U.S. Holocaust Memorial Museum 


http: / /www.ushmm.org/ 


i 


0.0% 


1 


0.1% 


UC Davis' History Project 


http: / /historyproject.ucdavis.ed 

u/ 


i 


0.0% 


1 


0.1% 


UC History Digital Archive 


http:/ /sunsite.berkeley.edu/uch 
istory/ 


i 


0.0% 


1 


0.1% 


UC Writing Institute 


http: //ucwrite.org/ 


i 


0.0% 


1 


0.1% 


UCSB Hull Digital Image Library (HDIL) 


http:/ / www.womst.ucsb.edu/ p 
rojects/hull/ 


i 


0.0% 


1 


0.1% 


US Coast Guard Lighthouses 


http:/ / www.uscg.mil/hq/g- 
cp/history/ h_lhindex.html 


i 


0.0% 


1 


0.1% 


US Department of Energy 


http:// www.energy.gov/ engine 
/content, do 


i 


0.0% 


1 


0.1% 


US Department of Labor 


http: / /www. dol.gov/ 


i 


0.0% 


1 


0.1% 


US Diplomatic Mission to Germany 


http:// www.usembassy.de/ usa 
/ garelations4555 .htm 


i 


0.0% 


1 


0.1% 


US Supreme Court 


http:// www.supremecourtus.go 

v/ 


i 


0.0% 


1 


0.1% 


Uncle Tom's cabin and American Culture 


http: / / www.iath.virginia.edu / u 
tc/ 


i 


0.0% 


1 


0.1% 


United Nations (UN) 


http://www.im.org/ 


i 


0.0% 


1 


0.1% 


| University of California Berkeley library German collections 


i 


0.0% 


1 


0.1% 


University of California eScholarship 
press 


http:// ark.cdlib.org/ ark:/13030 
/ ft038n99hg/ 


i 


0.0% 


1 


0.1% 


University of Edinburgh Walter Scott 
archive 


http: / / www.walterscott.lib.ed.a 
c.uk/ 


i 


0.0% 


1 


0.1% 


| University of Kansas' primary documents page 


i 


0.0% 


1 


0.1% 


University of Toronto archives 


http: / /www.library.utoronto.ca 
/utarms/ 


i 


0.0% 


1 


0.1% 


VERA - Virtual Electronic Resource 
Access 


http :// river .mit. edu/ mitlibweb 
/ FMPro?-db=RS_Items.fp5&- 
Lay=w 


i 


0.0% 


1 


0.1% 


Vanderbilt Television Archives 


http: //tvnews.vanderbilt.edu/ 


i 


0.0% 


1 


0.1% 


Vincent Ferraro's International Relations 
and Foreign Policy 


http:/ /www.mtholyoke.edu/ac 
ad/intrel/feros-pg.htm 


i 


0.0% 


1 


0.1% 
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Virtual Courseware 


http: / / vcourseware5.calstatela.e 
du/ 


i 


0.0% 


1 


0.1% 


Virtual Data Center 


http: / / thedata.org/ 


i 


0.0% 


1 


0.1% 


Virtual Jamestown 


http:/ /www.virtualjamestown.o 

rg/ 


i 


0.0% 


1 


0.1% 


Virtual Salt 


http: //virtualsalt.com/ 


i 


0.0% 


1 


0.1% 


Virtual Vaudeville 


http: / / www.virtualvaudeville.c 
om/ 


i 


0.0% 


1 


0.1% 


Visual Sourcebook for Chinese 
Civilization 


http:/ / depts.washington.edu/c 
hinaciv / 


i 


0.0% 


1 


0.1% 


Visual Thesaurus 


http : / / www .visualthesaurus.co 
m/ 


i 


0.0% 


1 


0.1% 


Vroma: A Virtual Community for 
Teaching and Learning Classic 


http : / / www . vroma. or g/ 


i 


0.0% 


1 


0.1% 


Washington Monthly 


http : / / www . washingtonmonthl 
y.com/ 


i 


0.0% 


1 


0.1% 


| Washington State University's online history course 


i 


0.0% 


1 


0.1% 


Web Archaeology - Levi Jordan 
Plantation 


http:/ / www.webarchaeology.co 
m 


i 


0.0% 


1 


0.1% 


Web Gallery of Art 


http : / / www .wga.hu/ index, htm 
1 


i 


0.0% 


1 


0.1% 


Webref (reference site) 


http://webref.org/ 


i 


0.0% 


1 


0.1% 


Westlaw 


http: / /web2.westlaw.com/ 


i 


0.0% 


1 


0.1% 


Whitney Museum 


http:/ /www .whitney.org/ 


i 


0.0% 


1 


0.1% 


Willkommen bei Sammelpunkt. 
Elektronisch archivierte Theorie 


http:// sammelpunkt.philo.at:80 
80/ 


i 


0.0% 


1 


0.1% 


Witcombe's art history materials on the 
Sweet Briar College 


http: / / witcombe.sbc.edu/ART 
HLinks.html 


i 


0.0% 


1 


0.1% 


Women's Budget Groups 


http:/ /www .wbg.org.uk/ 


i 


0.0% 


1 


0.1% 


World Health Organization 


http://www.who.int/en/ 


i 


0.0% 


1 


0.1% 


World History Center 


http : / / www .whc .neu.edu / 


i 


0.0% 


1 


0.1% 


World History Center - migration 


http://www.whc.neu.edu/ 


i 


0.0% 


1 


0.1% 


World History Connected 


http://worldhistoryconnected.p 

ress.uiuc.edu/ 


i 


0.0% 


1 


0.1% 


World War II Poster Collection 
(Northwestern University) 


http:/ / www. library .northwester 
n.edu/govpub/ collections/ wwi 
i- 


i 


0.0% 


1 


0.1% 


World of Stereoviews: US States 


http: / / www.worldofstereoview 
s.com/ USstatespagel .htm 


i 


0.0% 


1 


0.1% 


Worlds of Difference 


http:/ /homelands.org/ worlds/ 


i 


0.0% 


1 


0.1% 


Wright American Fiction 


http://www.letrs.indiana.edu/ 

web/w/wright2/ 


i 


0.0% 


1 


0.1% 


Zogby (poll) 


http://www.zogby.com/ 


i 


0.0% 


1 


0.1% 


| academic websites on writing/ pedagogy 


i 


0.0% 


1 


0.1% 


an online syllabus 


http : / / www .swarthmore . edu/ 

Humanities/kjohnsol/pictures/ 

paint 


i 


0.0% 


1 


0.1% 


chat rooms 




i 


0.0% 


1 


0.1% 


collocation/ corpus/ concordancing programs 


i 


0.0% 


1 


0.1% 


database - ABI/Inform (business database) 


i 


0.0% 


1 


0.1% 


database - American Periodical Series 


i 


0.0% 


1 


0.1% 


database - BIOSIS 




i 


0.0% 


1 


0.1% 


database - Chuyus 




i 


0.0% 


1 


0.1% 


database - Dialog 




i 


0.0% 


1 


0.1% 


| database - Dialog - Philosopher's Index 


i 


0.0% 


1 


0.1% 


database - Die Deutsche Bibliothek 
Database 


http:/ / dbf-opac.ddb.de/ 


i 


0.0% 


1 


0.1% 
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database - Dissertation Abstracts international 


i 


0.0% 


1 


0.1% 


| database - GALE databases 




i 


0.0% 


1 


0.1% 


(database - History of science, technology, and medicine 


i 


0.0% 


1 


0.1% 


| database - Humanities Databases 




i 


0.0% 


1 


0.1% 


| database - KV 55 Database (egyptology?) 


i 


0.0% 


1 


0.1% 


database - OCLC ArticleFirst 




i 


0.0% 


1 


0.1% 


database - STN 




i 


0.0% 


1 


0.1% 


database - UC Davis historical images 


http: //historyproject.ucdavis.ed 
u / imageapp .php 


i 


0.0% 


1 


0.1% 


database - anthropology 




i 


0.0% 


1 


0.1% 


database - historical images 




i 


0.0% 


1 


0.1% 


database - languages 




i 


0.0% 


1 


0.1% 


database - literature 




i 


0.0% 


1 


0.1% 


database - periodical 




i 


0.0% 


1 


0.1% 


| database - political science ressearch 


i 


0.0% 


1 


0.1% 


database - speech archives 




i 


0.0% 


1 


0.1% 


database - university 




i 


0.0% 


1 


0.1% 


dictionaries - Chinese - English 




i 


0.0% 


1 


0.1% 


digitized collections of painting, sculpture and installatio 


i 


0.0% 


1 


0.1% 


digitized images from departmental slide library 


i 


0.0% 


1 


0.1% 


images - Departmental digital image collection for history c 


i 


0.0% 


1 


0.1% 


images - digital image libraries 




i 


0.0% 


1 


0.1% 


images - online galleries 




i 


0.0% 


1 


0.1% 


images - online galleries for Nazism/ Holocaust 


i 


0.0% 


1 


0.1% 


images - university's digital image library 


i 


0.0% 


1 


0.1% 


lapl.org is LA public library site 


lapl.edu 


i 


0.0% 


1 


0.1% 


libraries - Huntington Library Online 




i 


0.0% 


1 


0.1% 


libraries - Kansas University (KU) 


http://www.lib.ku.edu/ 


i 


0.0% 


1 


0.1% 


libraries - UCB Pathfinder 


http: / / sunsite5.berkeley.edu:800 
0/ 


i 


0.0% 


1 


0.1% 


(libraries - UCSB Alexandria Digital Library 


i 


0.0% 


1 


0.1% 


libraries - UCSD Library Catalogue (Roger 


) 


i 


0.0% 


1 


0.1% 


libraries - UCSD Library Catalogue (SAGE 


) 


i 


0.0% 


1 


0.1% 


(libraries - University of Aberdeen Library (Scotland) 


i 


0.0% 


1 


0.1% 


libraries - archived collections 




i 


0.0% 


1 


0.1% 


libraries - audio/ digital sound libraries 




i 


0.0% 


1 


0.1% 


libraries - library reference collections 




i 


0.0% 


1 


0.1% 


(libraries - websites of national libraries 


i 


0.0% 


1 


0.1% 


list - ANE 




i 


0.0% 


1 


0.1% 


list - FLTEACH 




i 


0.0% 


1 


0.1% 


list - H-Afro-Am 




i 


0.0% 


1 


0.1% 


list - H-Amlnd (American Indian) 




i 


0.0% 


1 


0.1% 


list - H-Environment 


http://www.h- 
net. or g/ ~environ / 


i 


0.0% 


1 


0.1% 


list - H-German 




i 


0.0% 


1 


0.1% 


list - H-Polmeth 




i 


0.0% 


1 


0.1% 


list - H-Rhetor 




i 


0.0% 


1 


0.1% 


list - H-Teach 




i 


0.0% 


1 


0.1% 


list - H-World 




i 


0.0% 


1 


0.1% 


(list - Queatre (Quebec Theatre discussion) 


i 


0.0% 


1 


0.1% 


list - VICTORIA and its archives 


https:/ /listserv.indiana.edu/arc 
hives/victoria.html 


i 


0.0% 


1 


0.1% 


(list - WMST-L (Women's Studies Email list) 


i 


0.0% 


1 


0.1% 


list - WPA 




i 


0.0% 


1 


0.1% 


list - archived lists 




i 


0.0% 


1 


0.1% 


list- H-Film 




i 


0.0% 


1 


0.1% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


list- H-Urban 




i 


0.0% 


1 


0.1% 


my own audio 




i 


0.0% 


1 


0.1% 


my own blog - kim 


http: / / professorkim.blogspot.co 
m/ 


i 


0.0% 


1 


0.1% 


|my own department's Writing Center website 


i 


0.0% 


1 


0.1% 


my own digitized map collection 




i 


0.0% 


1 


0.1% 


my own primary achaeological data 




i 


0.0% 


1 


0.1% 


my own simulations 




i 


0.0% 


1 


0.1% 


my own website - Edlndex 


http:/ / www .pitt.edu/ -poole/ e 
dmenu.html 


i 


0.0% 


1 


0.1% 


myUCDavis portal 


http://my.ucdavis.edu/ 


i 


0.0% 


1 


0.1% 


news - historical newspapers online 




i 


0.0% 


1 


0.1% 


news online - Arabic 




i 


0.0% 


1 


0.1% 


news online - Indonesia 




i 


0.0% 


1 


0.1% 


news online - Italian 




i 


0.0% 


1 


0.1% 


news online - South Asia 




i 


0.0% 


1 


0.1% 


news online - international 




i 


0.0% 


1 


0.1% 


news online - scientific news 




i 


0.0% 


1 


0.1% 


oVID/ cinovid 


http: / /cinovid.org/ 


i 


0.0% 


1 


0.1% 


online - collections of Buddhist sutras 




i 


0.0% 


1 


0.1% 


(online - collections of historical documents 


i 


0.0% 


1 


0.1% 


online Russian magazines 




i 


0.0% 


1 


0.1% 


online bibliographies - for art history 




i 


0.0% 


1 


0.1% 


online books 




i 


0.0% 


1 


0.1% 


online collections - Victorian British Collections 


i 


0.0% 


1 


0.1% 


online collections of German medieval manuscripts 


i 


0.0% 


1 


0.1% 


online collections of images for Latin America 


i 


0.0% 


1 


0.1% 


online companion to Oxford Anthology of American Poetry 


i 


0.0% 


1 


0.1% 


online data - from Population Research Board 


i 


0.0% 


1 


0.1% 


online data - from child welfare websites 


i 


0.0% 


1 


0.1% 


online dictionaries - Buddhist 


i 


0.0% 


1 


0.1% 


online dictionaries - Michaelis (Portuguese) 


i 


0.0% 


1 


0.1% 


online discussion groups - Anglo Saxon 


i 


0.0% 


1 


0.1% 


online game communities and subcultures 


i 


0.0% 


1 


0.1% 


(online guides - "how to" 




i 


0.0% 


1 


0.1% 


online index - Hispanic american periodicals index 


i 


0.0% 


1 


0.1% 


online journals - American Psychology Association (APA) jour 


i 


0.0% 


1 


0.1% 


online journals - biological sciences 




i 


0.0% 


1 


0.1% 


| online language dictionary from another university 


i 


0.0% 


1 


0.1% 


online magazines 




i 


0.0% 


1 


0.1% 


online portals - for news 




i 


0.0% 


1 


0.1% 


online portals - for non-profits 




i 


0.0% 


1 


0.1% 


online resources - for books 




i 


0.0% 


1 


0.1% 


(online resources - government statistical resources 


i 


0.0% 


1 


0.1% 


online resources - historical info 




i 


0.0% 


1 


0.1% 


online resources - oral histories 




i 


0.0% 


1 


0.1% 


online simulations 




i 


0.0% 


1 


0.1% 


online social science abstracts 




i 


0.0% 


1 


0.1% 


online thesaurus 




i 


0.0% 


1 


0.1% 


| personal collection - a colleague's digital video 


i 


0.0% 


1 


0.1% 


search engines - Japanese 




i 


0.0% 


1 


0.1% 


streaming video 




i 


0.0% 


1 


0.1% 


(student guides on critical thinking and web use 


i 


0.0% 


1 


0.1% 


textbases- text archives 




i 


0.0% 


1 


0.1% 


the White House 


http:/ /www .whitehouse.gov/ 


i 


0.0% 


1 


0.1% 


university art museum websites 




i 


0.0% 


1 


0.1% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


usgvt.edu 


usgvt.edu 


i 


0.0% 


1 


0.1% 


website for Amelia Statistical Software 


http: / / gking.harvard.edu/ stats. 
shtml/#amelia 


i 


0.0% 


1 


0.1% 


websites - biblical commentary 




i 


0.0% 


1 


0.1% 


websites - African 




i 


0.0% 


1 


0.1% 


websites - African Diaspora Art 




i 


0.0% 


1 


0.1% 


websites - Dead Sea Scrolls 




i 


0.0% 


1 


0.1% 


(websites - Foreign Affairs department of East Asian states(K 


i 


0.0% 


1 


0.1% 


websites - French 




i 


0.0% 


1 


0.1% 


websites - German TV 




i 


0.0% 


1 


0.1% 


websites - German radio 




i 


0.0% 


1 


0.1% 


websites - Gothic architecture 




i 


0.0% 


1 


0.1% 


websites - Holy Grail 




i 


0.0% 


1 


0.1% 


websites - Japanese 




i 


0.0% 


1 


0.1% 


websites - Korean 




i 


0.0% 


1 


0.1% 


(websites - Kyoto and Tokyo National Museums 


i 


0.0% 


1 


0.1% 


| websites - Mary Magdalen 




i 


0.0% 


1 


0.1% 


| websites - Michigan State's sites on Africa 


i 


0.0% 


1 


0.1% 


websites - Paris Faculty of Medicine 




i 


0.0% 


1 


0.1% 


websites - Renaissance Art 




i 


0.0% 


1 


0.1% 


websites - Shakespeare 




i 


0.0% 


1 


0.1% 


(websites - South African resistance in art 


i 


0.0% 


1 


0.1% 


websites - Spanish 




i 


0.0% 


1 


0.1% 


websites - Templars 




i 


0.0% 


1 


0.1% 


websites - Turner's paintings 




i 


0.0% 


1 


0.1% 


websites - U.S. executive branch agencies 


i 


0.0% 


1 


0.1% 


websites - US/Canadian universities on teaching Indonesian 


i 


0.0% 


1 


0.1% 


websites - World War I 




i 


0.0% 


1 


0.1% 


websites - about travel 




i 


0.0% 


1 


0.1% 


websites - academic 




i 


0.0% 


1 


0.1% 


(websites - accessibility sites (508 compliance) 


i 


0.0% 


1 


0.1% 


websites - activists 




i 


0.0% 


1 


0.1% 


websites - advocacy and policy 




i 


0.0% 


1 


0.1% 


websites - art - images - Chicano art 




i 


0.0% 


1 


0.1% 


websites - artwork - religious artwork 




i 


0.0% 


1 


0.1% 


websites - biographies of women 




i 


0.0% 


1 


0.1% 


websites - coins 




i 


0.0% 


1 


0.1% 


websites - comedia 




i 


0.0% 


1 


0.1% 


websites - commercial websites related to course materials 


i 


0.0% 


1 


0.1% 


websites - conspiracy theories generally 


i 


0.0% 


1 


0.1% 


websites - designers/ design agencies 


i 


0.0% 


1 


0.1% 


websites - drama 




i 


0.0% 


1 


0.1% 


websites - electronic text centers 




i 


0.0% 


1 


0.1% 


(websites - for Western Civilization textbooks 


i 


0.0% 


1 


0.1% 


websites - government - US law 




i 


0.0% 


1 


0.1% 


websites - historical images 




i 


0.0% 


1 


0.1% 


| websites - index to traditional sources (books and articles) 


i 


0.0% 


1 


0.1% 


websites - legal history 




i 


0.0% 


1 


0.1% 


websites - links Camino de Santiago 




i 


0.0% 


1 


0.1% 


websites - links US Mexico Border 




i 


0.0% 


1 


0.1% 


websites - links on other people's/ organization's sites 


i 


0.0% 


1 


0.1% 


websites - links to Society of American Poets 


i 


0.0% 


1 


0.1% 


| websites - local government site 




i 


0.0% 


1 


0.1% 


websites - maintained by the Society of Shin Buddhism (in Ja 


i 


0.0% 


1 


0.1% 


websites - major university site for Chicano/ Latino studies 


i 


0.0% 


1 


0.1% 


websites - of Spanish textbooks 




i 


0.0% 


1 


0.1% 
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Response (site name or description) 


URL 


# respon- 
ses 


% respon- 
ses 


# people 


% people 


websites - of current organizations compiling data on voting 


i 


0.0% 


1 


0.1% 


websites - of departments 




i 


0.0% 


1 


0.1% 


websites - of laboratories 




i 


0.0% 


1 


0.1% 


websites - of organizations 




i 


0.0% 


1 


0.1% 


websites - of researchers 




i 


0.0% 


1 


0.1% 


websites - official EU websites 




i 


0.0% 


1 


0.1% 


websites - on house and senate 




i 


0.0% 


1 


0.1% 


websites - on legal codes 




i 


0.0% 


1 


0.1% 


websites - online translations of classical texts 


i 


0.0% 


1 


0.1% 


websites - personal webpages 




i 


0.0% 


1 


0.1% 


websites - personal, non-commercial on Indonesian short stor 


i 


0.0% 


1 


0.1% 


websites - political economy 




i 


0.0% 


1 


0.1% 


websites - presentation of visual material 


i 


0.0% 


1 


0.1% 


websites - public broadcasting 




i 


0.0% 


1 


0.1% 


websites - public interest 




i 


0.0% 


1 


0.1% 


websites - related to radios 




i 


0.0% 


1 


0.1% 


websites - resisting Vietnam 




i 


0.0% 


1 


0.1% 


websites - site that is a list of other sites (portal) 


i 


0.0% 


1 


0.1% 


websites - state government 




i 


0.0% 


1 


0.1% 


websites - state parks 




i 


0.0% 


1 


0.1% 


websites - that provide cultural or linguistic realia and in 


i 


0.0% 


1 


0.1% 


websites - university-based Anthropology 


i 


0.0% 


1 


0.1% 


websites - university-based Humanities 


i 


0.0% 


1 


0.1% 
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Appendix J: Faculty Survey Breakdown: Digital Resource Use in the 
Community College System 

Among the community colleges, it is possible to break down the results by the size of the 
institution and the population density of the community college district (urban, suburban, or 
rural). Because the Ns of each group are relatively small, we should interpret with caution. 

Note that all three sizes of community college (large, medium and small) occur in all three 
geographic regions (rural, suburban, and urban). (See Tables H.5 and H.6 in Appendix H.) 

Overall, instructors at both rural and suburban schools use more digital resources than those at 
urban schools. Suburban instructors are more enthusiastic about digital resources, and seem to 
be involved in more technologically advanced applications; for example, they are twice as likely 
as rural or urban faculty to use materials in the context of online lectures or online discussions. 
In general, suburban instructors experience the fewest barriers to digital resource use. 

Suburban instructors have the best access to classroom technology, servers, and scanners, and 
the best access to computers and high-speed connections for both themselves and their 
students. Suburban instructors also need less support for almost all tasks. It appears that 
suburban community colleges provide the best technical infrastructure and support to their 
teachers and students, and perhaps attract instructors who are more skilled and adept at using 
online materials. 

Rural institutions are also heavy users of digital resources. Rural instructors are the most likely 
to be motivated to use digital resources to gain access to materials that would otherwise be 
unavailable at their school. They also believe that digital resources improve their students' 
learning, by providing access to primary sources, helping teach information literacy, and 
providing examples of good and bad scholarship. 

Urban instructors are particularly concerned about their students' preparation and skills — the 
students' lack of access to computers and Internet connections, but also the students' inability to 
assess the credibility of online resources and their tendency to copy or plagiarize from online 
sources. Urban instructors need support with a broad range of activities— both technical 
activities, such as setting up the technical infrastructure, and intellectual, content-based 
activities, such as finding and organizing digital materials and assessing their credibility and 
appropriateness for teaching. Access to equipment is an issue in rural California, as well. As 
one instructor said, "At the community college level in rural California, there are just not 
enough computers." 

Within each geographic category, larger institutions tend to be lighter digital resource users. 
Larger schools are the most likely to have problems with access to technology, including 
classroom technology, scanners, and computers for both students and instructors. Instructors at 
larger institutions also express a greater need for support, particularly with technical needs. 
Perhaps resources and support may be spread quite thin at the larger community colleges. 
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Appendix K: Cluster Descriptions 



Table K.l: Cluster descriptions, part 1 





Institution 


Position 


Disciplines 


Type of course 


Age 


Gender 


Education 


Total Use 


Enthusiasm 


All 

N=608 


63% UC 
12% LAC 
24% CC 


66% tenure-track 
33% non-tenure-track 

79% full time 
21 % part time 


27% English/literature 
12% Foreign languages 
12% Art/architecture 
12% History 
11% Political science 
9% Anthro/archaeo 
4% Writing 

3% Ethnic/gender/cultural 
3% Geography 
2% Media/communication 


57% Seminar 
14% Lab 

43% Small lecture 
52% Medium lecture 
31% Large lecture 
22% Very large lecture 
5% Online course 
21% Hybrid course 
87%T raditional course 


49.2 ± 10.5 


48% Male 
52% Female 


2% AA/BA 
25% Master's 
72% Ph.D. 


33.6 ±14.4 


5. ±1.6 


Cluster 1 
N=60 


70% UC 
13% LAC 
15% CC 


73% tenure-track 
25% non-tenure-track 

79% full time 
21 % part time 


42% English/literature 
5% Foreign languages 
5% Art/architecture 
7% History 
17% Political science 
8% Anthro/archaeo 
3% Writing 

2% Ethnic/gender/cultural 
2% Geography 
2% Media/communication 


70% Seminar 
3% Lab 

32% Small lecture 
45% Medium lecture 
42% Large lecture 
23% Very large lecture 
2% Online course 
3% Hybrid course 
97%T raditional course 


53.8 ± 10.1 


67% Male 
33% Female 


3% AA/BA 
13% Master's 
83% Ph.D. 


14.4 ±7.6 


2.3 ±1.1 


Cluster 2 
N=1 06 


51% UC 
11% LAC 
37% CC 


60% tenure-track 
39% non-tenure-track 

78% full time 
22% part time 


42% English/literature 
11% Foreign languages 
5% Art/architecture 
10% History 
9% Political science 
4% Anthro/archaeo 
5% Writing 

8% Ethnic/gender/cultural 
1% Geography 
1% Media/communication 


45% Seminar 
11% Lab 

42% Small lecture 
53% Medium lecture 
25% Large lecture 
14% Very large lecture 
2% Online course 
6% Hybrid course 
92%T raditional course 


50.7 ± 10.3 


42% Male 
58% Female 


1% AA/BA 
38% Master's 
61% Ph.D. 


27.7 ±9.2 


4.1 ±1.3 
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Institution 


Position 


Disciplines 


Type of course 


Age 


Gender 


Education 


Total Use 


Enthusiasm 


All 

N=608 


63% UC 
12% LAC 
24% CC 


66% tenure-track 
33% non-tenure-track 

79% full time 
21 % part time 


27% English/literature 
12% Foreign languages 
12% Art/architecture 
12% History 
11% Political science 
9% Anthro/archaeo 
4% Writing 

3% Ethnic/gender/cultural 
3% Geography 
2% Media/communication 


57% Seminar 
14% Lab 

43% Small lecture 
52% Medium lecture 
31 % Large lecture 
22% Very large lecture 
5% Online course 
21% Hybrid course 
87%T raditional course 


49.2 ± 10.5 


48% Male 
52% Female 


2% AA/BA 
25% Master's 
72% Ph.D. 


33.6 ±14.4 


5. ±1.6 


Cluster 3a 
N=48 


69% UC 
27% LAC 
4 ± 20% CC 


65% tenure-track 
31 % non-tenure-track 

81 % full time 
19% part time 


35% English/literature 
13% Foreign languages 
6% Art/architecture 
6% History 
23% Political science 
4% Anthro/archaeo 
4% Writing 

0% Ethnic/gender/cultural 
0% Geography 
2% Media/communication 


71% Seminar 
2% Lab 

50% Small lecture 
60% Medium lecture 
23% Large lecture 
15% Very large lecture 
4% Online course 
17% Hybrid course 
83%T raditional course 


46.9 ± 10.2 


52% Male 
48% Female 


0% AA/BA 
13% Master's 
88% Ph.D. 


23.4 ± 8 


4.8 ±1.3 


Cluster 3b 
14=95 


72% UC 
9% LAC 
19%CC 


67% tenure-track 
32% non-tenure-track 

76% full time 
24% part time 


23% English/literature 
13% Foreign languages 
21 % Art/architecture 
11% History 
3% Political science 
13% Anthro/archaeo 
3% Writing 

2% Ethnic/gender/cultural 
2% Geography 
1% Media/communication 


69% Seminar 
13% Lab 

48% Small lecture 
48% Medium lecture 
40% Large lecture 
23% Very large lecture 
1% Online course 
14% Hybrid course 
96%T raditional course 


48.4 ± 9.6 


34% Male 
66% Female 


0% AA/BA 
25% Master's 
74% Ph.D. 


30.5 ± 9.3 


5.1 ±1.1 


Cluster 3c 
N=63 


81% UC 
10% LAC 
10%CC 


81 % tenure-track 
19% non-tenure-track 

90% full time 
10% part time 


27% English/literature 
13% Foreign languages 
5% Art/architecture 
19% History 
16% Political science 
6% Anthro/archaeo 
0% Writing 

5% Ethnic/gender/cultural 
5% Geography 
0% Media/communication 


63% Seminar 
5% Lab 

41% Small lecture 
57% Medium lecture 
32% Large lecture 
32% Very large lecture 
0% Online course 
25% Hybrid course 
83%T raditional course 


49.6 ± 10.8 


66% Male 
34% Female 


0% AA/BA 
10% Master's 
90% Ph.D. 


34.5 ±10.9 


5.4 ±1.2 
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Institution 


Position 


Disciplines 


Type of course 


Age 


Gender 


Education 


Total Use 


Enthusiasm 


All 

N=608 


63% UC 
12% LAC 
24% CC 


66% tenure-track 
33% non-tenure-track 

79% full time 
21 % part time 


27% English/literature 
12% Foreign languages 
12% Art/architecture 
12% History 
11% Political science 
9% Anthro/archaeo 
4% Writing 

3% Ethnic/gender/cultural 
3% Geography 
2% Media/communication 


57% Seminar 
14% Lab 

43% Small lecture 
52% Medium lecture 
31 % Large lecture 
22% Very large lecture 
5% Online course 
21% Hybrid course 
87%T raditional course 


49.2 ± 10.5 


48% Male 
52% Female 


2% AA/BA 
25% Master's 
72% Ph.D. 


33.6 ±14.4 


5. ±1.6 


Cluster 4a 
N=62 


76% UC 
8% LAC 
15%CC 


79% tenure-track 
18% non-tenure-track 

87% full time 
13% part time 


18% English/literature 
5% Foreign languages 
6% Art/architecture 
29% History 
10% Political science 
16% Anthro/archaeo 
0% Writing 

5% Ethnic/gender/cultural 
5% Geography 
2% Media/communication 


69% Seminar 
10% Lab 

40% Small lecture 
55% Medium lecture 
34% Large lecture 
45% Very large lecture 
0% Online course 
10% Hybrid course 
95%T raditional course 


47.9 ±9.5 


60% Male 
40% Female 


2% AA/BA 
13% Master's 
85% Ph.D. 


37.6 ±11.2 


6. ±1.2 


Cluster 4b 
N=36 


56% UC 
8% LAC 
31% CC 


53% tenure-track 
44% non-tenure-track 

60% full time 
40% part time 


25% English/literature 
14% Foreign languages 
14% Art/architecture 
6% History 
14% Political science 
11% Anthro/archaeo 
6% Writing 

0% Ethnic/gender/cultural 
3% Geography 
3% Media/communication 


39% Seminar 
22% Lab 

39% Small lecture 
42% Medium lecture 
19% Large lecture 
14% Very large lecture 
3% Online course 
17% Hybrid course 
86%T raditional course 


46.8 ± 12.3 


36% Male 
64% Female 


17% AA/BA 
28% Master's 
56% Ph.D. 


42.3 ±10.1 


5.8 ± 1. 


Cluster 4c 
14=68 


62% UC 
18% LAC 
19% CC 


66% tenure-track 
34% non-tenure-track 

81 % full time 
19% part time 


10% English/literature 
21% Foreign languages 
26% Art/architecture 
7% History 
9% Political science 
9% Anthro/archaeo 
1% Writing 

4% Ethnic/gender/cultural 
6% Geography 
0% Media/communication 


53% Seminar 
35% Lab 

44% Small lecture 
51% Medium lecture 
37% Large lecture 
25% Very large lecture 
0% Online course 
50% Hybrid course 
69%T raditional course 


48.5 ±9.7 


51% Male 
49% Female 


4% AA/BA 
32% Master's 
62% Ph.D. 


48.6 ±11.1 


6.3 ±1 
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Institution 


Position 


Disciplines 


Type of course 


Age 


Gender 


Education 


Total Use 


Enthusiasm 


All 

N=608 


63% UC 
12% LAC 
24% CC 


66% tenure-track 
33% non-tenure-track 

79% full time 
21 % part time 


27% English/literature 
12% Foreign languages 
12% Art/architecture 
12% History 
11% Political science 
9% Anthro/archaeo 
4% Writing 

3% Ethnic/gender/cultural 
3% Geography 
2% Media/communication 


57% Seminar 
14% Lab 

43% Small lecture 
52% Medium lecture 
31 % Large lecture 
22% Very large lecture 
5% Online course 
21% Hybrid course 
87%T raditional course 


49.2 ± 10.5 


48% Male 
52% Female 


2% AA/BA 
25% Master's 
72% Ph.D. 


33.6 ±14.4 


5. ±1.6 


Cluster 5a 
N=48 


46% UC 
8% LAC 
46% CC 


44% tenure-track 
54% non-tenure-track 

77% full time 
23% part time 


19% English/literature 
13% Foreign languages 
8% Art/architecture 
6% History 
13% Political science 
13% Anthro/archaeo 
13% Writing 

2% Ethnic/gender/cultural 
0% Geography 
6% Media/communication 


42% Seminar 
27% Lab 

54% Small lecture 
52% Medium lecture 
17% Large lecture 
10% Very large lecture 
13% Online course 
52% Hybrid course 
67%T raditional course 


47.5 ± 10.7 


21% Male 
79% Female 


2% AA/BA 
35% Master's 
63% Ph.D. 


41.4 ±10.1 


5.7 ±1.3 


Cluster 5b 
N=22 


14% UC 
5% LAC 
73%CC 


55% tenure-track 
41 % non-tenure-track 

77% full time 
23% part time 


14% English/literature 
5% Foreign languages 
23% Art/architecture 
18% History 
5% Political science 
5% Anthro/archaeo 
5% Writing 

0% Ethnic/gender/cultural 
9% Geography 
5% Media/communication 


27% Seminar 
18% Lab 

23% Small lecture 
73% Medium lecture 
36% Large lecture 
14% Very large lecture 
68% Online course 
64% Hybrid course 
86%T raditional course 


50.2 ± 14.7 


55% Male 
45% Female 


5% AA/BA 
59% Master's 
36% Ph.D. 


58.7 ±11 


6.8 ±0.4 
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Table K.2: Cluster descriptions, part 2 





Personal 

Collection 


What do you use? 


How do you use it? 


Why do or don't you use it? 


What are your 
barriers? 


What support do you 
need? 


All 

N=608 


3.4 ±1.4 


Genl purp/references 0.06 ± 0.98 
Images/AV 0.03 ±1.02 
Historical/primary 0.03 ± 0.98 
News/data 0.02 ± 1 .02 
Discussions/curricular 
0.02 ±1.03 


Projects 0.07 ± 1 .01 
Post to web 0.04 ± 1 .0 
Online class 
0.00 ±1.01 
In class 0.06 ±0.99 


Improve learning 0.07 ± 0.99 
Expectations/reputation -0.02 ± 0.96 
Inappropriateness 0.02 ± 1 .02 
Concerns about interpretation 
0.01 ± 0.98 

Time/convenience/access -0.01 ± 0.96 
Teach info literacy 0.03 ± 1 .0 
Make info public 0.04 ± 0.98 
Use free materials -0.04 ± 0.91 


Finding 0.02 ± 1.0 
Institution's equip 
0.00 ± 0.97 

Personal equip -0.01 ± 0.97 
Availability 0.03 ±0.98 
Student equip 0.01 ± 1.01 


Technical 0.03 ±0.96 
Intellectual -0.01 ± 0.98 


Cluster 1 
N=60 


1.9 ± 1 


Genl purp/references - 
0.80 ± 0.70 

Images/AV -0.74 ± 0.59 
Historical/primary -0.67 ± 0.55 
News/data -0.38 ± 0.60 
Discussions/curricular - 
0.13 ±0.62 


Projects -0.65 ± 0.54 
Post to web - 
0.71 ±0.41 
Online class - 
0.01 ± 0.32 
In class -0.90 ± 0.64 


Improve learning -1 .66 ± 0.64 
Expectations/reputation -0.41 ± 0.50 
Inappropriateness 0.21 ± 0.97 
Concerns about interpretation 
0.24 ± 1 .07 

Time/convenience/access -0.38 ± 0.76 
Teach info literacy -0.41 ± 0.61 
Make info public -0.33 ± 0.53 
Use free materials -0.22 ± 1.14 


Finding 0.34 ±1.09 
Institution's equip — 

0.28 ±1.01 

Personal equip 0.20 ± 1.05 
Availability 0.48 ± 1.21 
Student equip -0.37 ± 0.89 


Technical —0.26 ± 1.18 
Intellectual -0.07 ± 0.92 


Cluster 2 
N=1 06 


2.4 ±1.1 


Genl purp/references - 
0.32 ± 0.88 

Images/AV 0.12 ±0.78 
Historical/primary -0.48 ± 0.71 
News/data 0.32 ±0.95 
Discussions/curricular — 

0.48 ± 0.66 


Projects -0.03 ± 0.87 
Post to web - 
0.62 ± 0.72 
Online class - 
0.24 ± 0.44 
In class -0.02 ± 0.81 


Improve learning -0.17 ± 0.68 
Expectations/reputation 0.23 ± 0.95 
Inappropriateness 0.03 ± 0.91 
Concerns about interpretation 
0.15 ±1.05 

Time/convenience/access -0.52 ± 0.75 
Teach info literacy 0.31 ± 1.0 
Make info public -0.44 ± 0.88 
Use free materials -0.11 ±0.94 


Finding 0.68 ±0.92 
Institution's equip 
0.09 ± 1.01 

Personal equip 0.18 ± 1.03 
Availability -0.20 ±0.93 
Student equip -0.06 ± 1 .04 


Technical 0.12 ±0.80 
Intellectual 0.44 ± 0.96 


Cluster 

3a 

N=48 


2.5 ±1 


Genl purp/references 0.30 ± 0.81 
Images/AV -1.13 ±0.68 
Historical/primary -0.41 ± 0.68 
News/data -0.20 ± 0.95 
Discussions/curricular 
0.46 ± 0.75 


Projects -0.07 ± 0.82 
Post to web - 
0.02 ± 0.78 
Online class - 
0.08 ± 0.53 
In class -1.07 ±0.77 


Improve learning -0.36 ± 0.81 
Expectations/reputation -0.19 ± 1.12 
Inappropriateness 0.51 ± 0.75 
Concerns about interpretation 
0.01 ± 0.98 

Time/convenience/access 0.17 ±0.93 
Teach info literacy —0.1 8 ± 1.05 
Make info public 0.21 ± 0.90 
Use free materials 0.17 ±0.79 


Finding 0.28 ±0.77 
Institution's equip 
0.07 ± 0.97 

Personal equip 0.11 ± 1.01 
Availability 0.04 ± 0.86 
Student equip -0.23 ± 0.87 


Technical 0.49 ±0.63 
Intellectual -0.11 ± 1.0 
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Personal 

Collection 


What do you use? 


How do you use it? 


Why do or don't you use it? 


What are your 
barriers? 


What support do you 
need? 


All 

N=608 


3.4 ±1.4 


Genl purp/references 0.06 ± 0.98 
Images/AV 0.03 ±1.02 
Historical/primary 0.03 ± 0.98 
News/data 0.02 ± 1 .02 
Discussions/curricular 
0.02 ±1.03 


Projects 0.07 ± 1 .01 
Post to web 0.04 ± 1 .0 
Online class 
0.00 ±1.01 
In class 0.06 ±0.99 


Improve learning 0.07 ± 0.99 
Expectations/reputation -0.02 ± 0.96 
Inappropriateness 0.02 ± 1 .02 
Concerns about interpretation 
0.01 ± 0.98 

Time/convenience/access -0.01 ± 0.96 
Teach info literacy 0.03 ± 1 .0 
Make info public 0.04 ± 0.98 
Use free materials -0.04 ± 0.91 


Finding 0.02 ± 1.0 
Institution's equip 
0.00 ± 0.97 

Personal equip -0.01 ± 0.97 
Availability 0.03 ±0.98 
Student equip 0.01 ± 1.01 


Technical 0.03 ±0.96 
Intellectual -0.01 ± 0.98 


Cluster 

3b 

N=95 


3.6 ±1.1 


Genl purp/references 0.30 ± 0.77 
Images/AV 0.24 ±0.68 
Historical/primary 0.16 ±0.77 
News/data -0.47 ± 0.90 
Discussions/curricular — 

0.49 ± 0.81 


Projects —0.10 ± 0.91 
Post to web 
0.25 ± 0.99 
Online class - 
0.31 ± 0.52 
In class 0.27 ± 0.83 


Improve learning 0.35 ± 0.67 
Expectations/reputation 0.51 ± 1 .06 
Inappropriateness 0.31 ± 0.90 
Concerns about interpretation 
0.35 ± 0.98 

Time/convenience/access 0.00 ± 0.96 
Teach info literacy —0.30 ± 1.03 
Make info public -0.07 ± 0.86 
Use free materials 0.27 ± 0.74 


Finding 0.36 ±0.84 
Institution's equip 
0.16 ±0.94 

Personal equip 0.15 ± 1.20 
Availability 0.29 ±0.94 
Student equip 0.19 ±0.94 


Technical 0.57 ± 0.54 
Intellectual 0.18 ±0.92 


Cluster 

3c 

N=63 


3.9 ±1.2 


Genl purp/references 0.25 ± 0.90 
Images/AV -0.62 ±0.78 
Historical/primary 0.55 ± 0.97 
News/data 0.01 ±1.11 
Discussions/curricular 
0.08 ± 0.77 


Projects -0.11 ± 0.90 
Post to web 
1.12 ±0.86 
Online class - 
0.45 ±0.53 
In class -0.38 ± 0.83 


Improve learning 0.07 ± 0.89 
Expectations/reputation -0.18 ± 0.78 
Inappropriateness 0.03 ±1.10 
Concerns about interpretation - 
0.18 ±0.81 

Time/convenience/access 0.55 ± 0.83 
Teach info literacy -0.39 ± 0.85 
Make info public 0.19 ± 1.09 
Use free materials —0.14 ± 0.94 


Finding -0.14 ±0.83 
Institution's equip 
0.04 ±0.89 

Personal equip -0.27 ± 0.84 
Availability 0.30 ±0.77 
Student equip 0.05 ± 1.04 


Technical —0.29 ± 1.0 
Intellectual -0.19 ± 0.82 


Cluster 

4a 

N=62 


4.3 ±1 


Genl purp/references - 
0.43 ± 0.91 

Images/AV -0.03 ± 0.74 
Historical/primary 1 .20 ± 0.94 
News/data 0.06 ± 1 .07 
Discussions/curricular - 
0.24 ± 0.82 


Projects -0.48 ± 0.88 
Post to web - 
0.05 ±1.08 
Online class - 
0.23 ± 0.68 
In class 1.05 ±0.54 


Improve learning 0.73 ±0.72 
Expectations/reputation -0.54 ± 0.74 
Inappropriateness 0.02 ±1.17 
Concerns about interpretation - 
0.50 ± 0.89 

Time/convenience/access -0.89 ± 0.76 
Teach info literacy -0.20 ± 0.86 
Make info public 0.13 ± 0.87 
Use free materials 0.02 ± 0.88 


Finding -0.58 ±0.98 
Institution's equip — 

0.05 ± 0.97 

Personal equip -0.09 ± 0.89 
Availability 0.02 ± 1.03 
Student equip -0.05 ± 1 .05 


Technical —0.13 ± 1.08 
Intellectual -0.56 ± 0.90 
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Personal 

Collection 


What do you use? 


How do you use it? 


Why do or don't you use it? 


What are your 
barriers? 


What support do you 
need? 


All 

N=608 


3.4 ±1.4 


Genl purp/references 0.06 ± 0.98 
Images/AV 0.03 ±1.02 
Historical/primary 0.03 ± 0.98 
News/data 0.02 ± 1 .02 
Discussions/curricular 
0.02 ±1.03 


Projects 0.07 ± 1 .01 
Post to web 0.04 ± 1 .0 
Online class 
0.00 ±1.01 
In class 0.06 ±0.99 


Improve learning 0.07 ± 0.99 
Expectations/reputation -0.02 ± 0.96 
Inappropriateness 0.02 ± 1 .02 
Concerns about interpretation 
0.01 ± 0.98 

Time/convenience/access -0.01 ± 0.96 
Teach info literacy 0.03 ± 1 .0 
Make info public 0.04 ± 0.98 
Use free materials -0.04 ± 0.91 


Finding 0.02 ± 1.0 
Institution's equip 
0.00 ± 0.97 

Personal equip -0.01 ± 0.97 
Availability 0.03 ±0.98 
Student equip 0.01 ± 1.01 


Technical 0.03 ±0.96 
Intellectual -0.01 ± 0.98 


Cluster 

4b 

N=36 


3.3 ±1.4 


Genl purp/references 0.74 ± 0.89 
Images/AV 0.48 ±1.04 
Historical/primary -0.32 ± 0.83 
News/data 0.66 ±0.97 
Discussions/curricular — 

0.54 ± 0.85 


Projects 0.62 ± 0.91 
Post to web - 
0.67 ±0.79 
Online class - 
0.28 ±0.65 
In class 0.61 ± 0.66 


Improve learning 0.54 ± 0.78 
Expectations/reputation -0.43 ± 0.69 
Inappropriateness -0.37 ± 0.94 
Concerns about interpretation - 
0.10 ±0.78 

Time/convenience/access 0.64 ± 0.79 
Teach info literacy 0.60 ± 0.96 
Make info public -0.86 ± 0.58 
Use free materials 0.15 ±0.85 


Finding -0.60 ±0.57 
Institution's equip 
0.12 ± 1.13 

Personal equip 0.09 ± 1 .04 
Availability 0.05 ±0.91 
Student equip -0.09 ± 1 .06 


Technical -0.01 ±0.96 
Intellectual -0.73 ± 0.65 


Cluster 

4c 

N=68 


4.5 ±0.7 


Genl purp/references 0.37 ± 1 .01 
Images/AV 0.99 ±0.97 
Historical/primary 0.37 ± 0.91 
News/data 0.12 ± 1.14 
Discussions/curricular 
0.49 ±1.22 


Projects 1 .01 ± 1 .0 
Post to web 
0.59 ±0.81 
Online class - 
0.23 ± 0.88 
In class 0.80 ±0.75 


Improve learning 0.62 ± 0.85 
Expectations/reputation -0.24 ± 0.81 
Inappropriateness -0.21 ±1.11 
Concerns about interpretation - 
0.35 ±0.81 

Time/convenience/access 0.36 ± 0.89 
Teach info literacy 0.50 ± 1.0 
Make info public 0.72 ± 0.87 
Use free materials -0.02 ± 0.88 


Finding -0.60 ±0.87 
Institution's equip — 

0.10 ±0.82 

Personal equip -0.22 ± 0.70 
Availability -0.11 ± 0.90 
Student equip 0.08 ± 1.04 


Technical -0.19 ±0.94 
Intellectual -0.11 ± 0.96 


Cluster 

5a 

N=48 


4. ±0.9 


Genl purp/references 0.68 ± 0.86 
Images/AV 0.24 ±1.07 
Historical/primary -0.43 ± 0.85 
News/data 0.14 ±0.92 
Discussions/curricular 
1.09 ±1.05 


Projects 0.64 ± 0.94 
Post to web 
0.31 ±0.86 
Online class 
1.12 ± 1.33 
In class —0.1 5 ± 1.0 


Improve learning 0.34 ± 0.68 
Expectations/reputation 0.40 ± 0.92 
Inappropriateness -0.18 ± 0.92 
Concerns about interpretation 
0.40 ± 0.90 

Time/convenience/access 0.50 ± 0.68 
Teach info literacy 0.40 ± 0.87 
Make info public 0.63 ± 0.97 
Use free materials -0.46 ± 0.86 


Finding -0.08 ±0.77 
Institution's equip — 

0.12 ±0.91 

Personal equip -0.26 ± 0.77 
Availability -0.30 ± 0.93 
Student equip 0.46 ± 0.98 


Technical —0.22 ± 1.11 
Intellectual 0.46 ± 0.99 
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Personal 

Collection 


What do you use? 


How do you use it? 


Why do or don't you use it? 


What are your 
barriers? 


What support do you 
need? 


All 

N=608 


3.4 ±1.4 


Genl purp/references 0.06 ± 0.98 
Images/AV 0.03 ±1.02 
Historical/primary 0.03 ± 0.98 
News/data 0.02 ± 1 .02 
Discussions/curricular 
0.02 ±1.03 


Projects 0.07 ± 1 .01 
Post to web 0.04 ± 1 .0 
Online class 
0.00 ±1.01 
In class 0.06 ±0.99 


Improve learning 0.07 ± 0.99 
Expectations/reputation -0.02 ± 0.96 
Inappropriateness 0.02 ± 1 .02 
Concerns about interpretation 
0.01 ± 0.98 

Time/convenience/access -0.01 ± 0.96 
Teach info literacy 0.03 ± 1 .0 
Make info public 0.04 ± 0.98 
Use free materials -0.04 ± 0.91 


Finding 0.02 ± 1.0 
Institution's equip 
0.00 ± 0.97 

Personal equip -0.01 ± 0.97 
Availability 0.03 ±0.98 
Student equip 0.01 ± 1.01 


Technical 0.03 ±0.96 
Intellectual -0.01 ± 0.98 


Cluster 

5b 

N=22 


4.9 ±0.4 


Genl purp/references 0.24 ± 1 .0 
Images/AV 1.25 ±0.83 
Historical/primary 0.50 ± 0.92 
News/data 0.56 ± 1 .02 
Discussions/curricular 
1.72 ±0.96 


Projects 0.48 ± 1.17 
Post to web 
0.54 ± 0.33 
Online class 
3.36 ± 0.87 
In class 0.42 ±0.53 


Improve learning 0.75 ± 0.59 
Expectations/reputation 0.43 ± 1.19 
Inappropriateness -1.11 ±0.93 
Concerns about interpretation - 
0.25 ± 1 .08 

Time/convenience/access 0.51 ± 0.84 
Teach info literacy 0.48 ± 1 .03 
Make info public 0.90 ± 1.0 
Use free materials -0.22 ± 0.91 


Finding -0.72 ±0.84 
Institution's equip 
0.05 ± 1.16 

Personal equip -0.40 ± 0.45 
Availability -0.77 ± 0.63 
Student equip 0.25 ± 1.02 


Technical -0.36 ± 1.01 
Intellectual -0.01 ± 1.01 
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Appendix L: Literature Review on User Profiling 

Prepared by Sarah Ellinger 

This literature review was undertaken to seek answers to the following questions: 

• What are the theoretical bases for user profiling? Ideally, what should it accomplish? 

• What are the current practices in user profiling? 

Alan Cooper and abstract profiling: Rather than depending on coarse notions of market segments. 
Cooper (2004) outlines a notion of "persona", a "precise description of our user and what he 
wishes to accomplish", as a formative evaluation tool — that is, a tool which is used in the 
creation of a product. A persona does not represent a particular user, but should be precisely 
envisioned down to a name, skills and an end goal; since one of the greatest values of a persona 
is its power to suggest extreme situations, it is more desirable that the persona be precise than it 
be an accurate or likely case. 125 Cooper strongly distinguishes between goals (end results) and 
tasks (the process by which the result is reached). Users are often unable to accurately describe 
their goals, especially personal goals (i.e., not wanting to feel embarrassed at one's inability to 
use a product). Once designers have created the relevant personas (Cooper recommends that a 
product have no more than three), they create "scenarios", walking their personas through 
possible use situations to identify constraints and shortcomings. 

Web services and personal profiling: While the Cooper model (abstract profiling) has thrived 
mostly in discussions of how to design new products, there is a parallel notion of user profiling 
in the computer science literature relating to user profiling as a tool for resources currently in 
use. The focus of this literature is automated user profiling, in which systems collect implicit 
data about users without their explicit participation, either for the purposes of analysis or in 
order to dynamically "personalize" the site for that user. All suggested models of personal 
profiling operate by collecting and weighting information such as cookies, search keywords, 
bookmarks, browser history, time spent at a page, navigation patterns and scrolling behavior. 
Digital privacy advocates such as Roger Clarke (1994) have said that personal profiling is "a 
potentially threatening, demeaning, and perhaps socially dangerous phenomenon." 

Current practice in digital cultural and educational resources: The Alice Grant Consulting (2003b) 
report on current evaluation practice in digital cultural collections found that 78% of 
responding institutions either used user profiling as a means of evaluation or (more commonly) 
worked from an informal model of their intended audience. The report suggests that there is a 
need for a standard of user profiling for cultural heritage and similar institutions; however, the 
profiles listed are based on descriptions of qualities rather than goals, as Cooper suggests. 

Other literature on digital libraries supports the conclusion that user profiling is not a common 
approach to designing cultural/education digital resources. There is a body of literature that 
argues that profiling should be adopted by libraries, but not outlining any more specialized 
principles that would be helpful. 



125 Cooper seems to take it for granted that the organization behind the product itself has precise and measurable 
goals (i.e., turning a profit). Many digital collections, especially the less formal ones, may have a much hazier 
mission. 
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Gary Marchionini and user/task taxonomies: Marchionini's user type/ task taxonomy was first 
outlined in his book. Information Seeking In Electronic Environments (1997), and was revised in 
Hert and Marchionini (1997) and Marchionini (2000). The taxonomy attempts to unify 
information about users' individual characteristics with information about their tasks and their 
information environments (Hert and Marchionini, 1997). However, it is worth noting that 
Marchionini does not describe any new approach to gathering information user's individual 
characteristics, but rather uses a mixture of common methodologies (surveys, interviews, 
observations), institutional divisions (university scholars, K-12 teachers), and self-reported 
categories to establish the primary user types which are then further granulated by the 
taxonomy. 126 Marchionini's 1997 version breaks down users by their 

• Personal attributes (physical, cognitive and social) 

• Prior experience in the relevant domain of knowledge 

• Experience in using the relevant information technology. 

Tasks, in contrast, are broken down by 

• Complexity (number of facets, amount of abstraction) 

• Specificity (how able or confident the user is to evaluate the accuracy and completeness 
of results) 

• Quantity (the amount of information expected) 

• Criticality (how important it is that the need be met) 

• Timeliness (how much time the user is willing to spend). 

Marchionini suggests that it is possible but impractical to define each attribute on a scale and 
create a complete matrix to populate with example cases. Instead he combines these dimensions 
into: 

• Motivation - the personal situation that brings a user to the resource, including quantity, 
criticality, and timeliness 

• Domain knowledge - related to the particular need, including knowledge of the subject 
area 

• Library system knowledge - including information technology 

• Focus - combining complexity, quantity and specificity 

• Time allocated - combining timeliness and criticality. 

Taking a metaphor from linguistics, Hert and Marchionini (1997) suggest describing tasks 
according to three dimensions: the pragmatic (what is the situation of the query), the semantic 
(what is the meaning of the query), and the syntagmatic (what is the expression of the query). 



126 



An example of a taxonomy in use is given in Marchionini 2000, analyzing users of the National Digital Library. 
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Carol Tenopir: Carol Tenopir (2003) argues for a more broadly focused profile including 
information about both tasks and users, and recommends the following six attributes: 

• Status 

• Discipline 

• Task 

• Institution 

• Age 

• Gender 
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Appendix M: Web TLA Software Package Comparison 

A number of software packages already exist for performing TLA. Among the approximately 
thirty-five free options, we evaluated the three that seem most viable. We compared them with 
three of the most widely used commercial packages, and with our own in-house system. 

Free tools: 

• AWStats (http:/ / awstats.sourceforge.net/): Of the free options, AWStats is the most 
full-featured. It is awkward to configure (it requires editing a text configuration file) 
and it is not too powerful when compared with some commercial options, but it is 
decent, frequently maintained, and produces good numbers. 

• Analog (http:/ /www.analog.cx/): Analog is maintained at the same rate as AWStats 
but seeks to address a different mode of use. Because Analog is frequently paired with a 
separate reporting tool, "Report Magic," the output looks sparse unless it has been re- 
processed by this tool. Analog is able to stand on its own, but on its own, it is inferior to 
AWStats. 

• Webalizer (http: / / www.mrunix.net/webalizer/): Webalizer hasn't been updated since 
2002, its numbers may be questionable, and its reports could be more significant. 
Nevertheless, it does something, and it does it well enough that it is still widely used. 
Again, configuration is cumbersome. 

Inexpensive commercial tool: 

• Wusage (http: / / www.boutell.com/ wusage/): Wusage offers few features that are not 
offered by free packages, but it comes with commercial support. It is cheap enough to 
be affordable for any project, and the availability of support may be beneficial. Because 
Wusage is not open-source, it is not possible to modify or customize its analyses or to 
verify its calculations (although this is unlikely to be a problem for most applications). 

High-end commercial tools: 

• Urchin (http: / / www.urchin.com/): In many ways. Urchin is a midpoint between 
Wusage and WebTrends. Urchin presents a similar range of reports as Wusage or free 
tools, but its strength lies in its very usable GUI. Configuration is still a challenge (as 
with Wusage), but the overall experience may be more user-friendly. As with 
WebTrends, Urchin can be configured for specific, customized reports. 

• WebTrends (http:/ /www.webtrends.com/): WebTrends is a very complex system that 
takes some getting used to. However, it is very powerful (legitimately addressing a 
range from hobby sites through international operations with multiple server rooms) 
and simplifies a wide range of marketing-related analyses. WebTrends may not be 
flexible enough to tackle an unlimited and arbitrary set of questions (such as some that 
are not financial in nature), but it will do a lot right out of the box. Despite its high cost, 
WebTrends may still be cheaper than coding a system in-house, depending on the 
specific situation. 
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Our in-house system: 

• In-house + SAS (http:/ / digitalresourcestudy.berkeley.edu): Our custom-developed 
system used a series of Perl scripts to preprocess the raw logs, loaded the data into a 
relational database (PostgreSQL), and calculated statistics using SAS. Like AWStats, 
Analog, and Webalizer, this package is potentially open-source, meaning the calculation 
of each number can be traced. The system is by far the most powerful of the tools we 
evaluated, due to its heavy dependence on PostgreSQL and SAS, both of which offer 
great data transparency and are widely used for manipulating large data sets. However, 
the system has not been packaged for re-distribution, and it is quite complicated to 
install without our help. 



http:/ / digitalresourcestudy.berkeley.edu 



10-107 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



Table M.l: TLA software package comparison 



Feature 


Wusage 


Webtrends 


Urchin 


AWStats 


Analog 


Webalizer 


CSHE + SAS 


Version 


8.0 P50 (9/16/04) 


7.0 (2005) 


6.0 ASP 127 
(9/2005) 


6.4 (3/16/05) 


6.0 (12/19/04) 


2.01-10 (4/16/02) 


0.064, 0.251 (2005) 


Source code 
available 


No 


No 


No 


Yes 


Yes 


Yes 


Maybe 


Price/ license 


$25 to $100 for 
academic use 


$10,000 for 20M 
annual page 
views 


$200/ (100k page 
views/mo)+$100 
/1M 


Free/GPL 


Free/GPL 


Free/GPL 


Unknown/ BSD? 


Requires tracking 
server/ service 128 


No 


Yes, but a hosted 
solution is 
available that 
requires no 
tracking server. 


No - the web 
server can double 
for this purpose. 
Also, a hosted 
solution is 
available. 


No 


No 


No 


No 


Requires 
JavaScript + 
cookies 129 


Yes 


Yes 


Yes 


No 


No 


No 


No 


Process load 
balanced logs 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Untested, but 
ought to work 


Report session 
duration 


No 


Yes 


Yes 


Yes 


No 


No 


Yes 


Report 

countries 130 


DNS 


DNS or geo- 
location 


DNS, possibly 
geo-location 


DNS or geo- 
location 


DNS 


DNS 


DNS 


Report hosts 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Report/ filter 
robots 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Yes 



127 We were unable to evaluate the newest version of Urchin first-hand. Our analysis is based on experience with a previous version and company literature about the 
newer release. 

128 A tracking server is a machine that records a log of human website activity in parallel with the web server's own log. Whereas the web server log records all web-server 
activity (including bots), a tracking server records events generated by JavaScript code, which is more specific to human behaviors. 

129 Requiring JavaScript implies that the website must be modified to integrate with the analysis software. 

130 Assuming a country report is available, what method was used to determine countries? DNS implies a method based on DNS lookups, and geolocation implies the use 
of a large index of all IP addresses and their likely country of origin 
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Feature 


Wusage 


Webtrends 


Urchin 


AWStats 


Analog 


Webalizer 


CSHE + SAS 


Report/ filter 
worms 


No 


Unknown 


Unknown 


Yes 


No 


No 


Possible with 
SAS regular 
expressions 


Report days of 
week 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Report most 
frequently 
viewed pages 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Report entry/ exit 
pages 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Possible with 
SAS/SQL 


Report pages by 
directory 


Yes 


Yes 


Yes 


No 


Yes 


No 


Possible with 
SAS regexp 


Report file types 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Report search 
engines used 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Report keywords 
used 


Yes 


Yes 


Yes 


Yes 


Partially 


Partially 


Yes 


Report referring 
web page 
with/without 
query 


Partially 


Yes 


Yes 


Yes 


No 


Partially 


Yes 


Report HTTP 
errors 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


No 131 


Custom analysis 


No 


Yes 


Yes 


No 


No 


No 


Yes 


Export 

aggregated data 


Yes (html) 


Yes (html. Word, 
CSV) 


Yes (html), likely 
others 


Yes (html, xml, 
txt) 


Yes (html, txt) 


Yes (html, txt) 


Yes (tab) 


Export 

observations 132 


No 


Maybe 


Maybe 


No 


No 


No 


Yes (RDBMS) 



131 Errors are filtered and attributed to a non-existent user rather than being tagged. 

132 That is, will the software export data that is meaningful to manipulate with a third-party tool toward any useful end? Exporting aggregated results will clearly not yield 
much in terms of data reuse, and the raw logs aren't appropriate to be loaded into Excel, for example, so does the package offer a happy medium? 
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Feature 


Wusage 


Webtrends 


Urchin 


AWStats 


Analog 


Webalizer 


CSHE + SAS 


Anonymization 

133 


No 


No 


No 


No 


No 


No 


Yes 


Barriers to entry 


Moderate: 

inexpensive 

license, 

commercial 

support 

available, 

requires 

sysadmin to 

install. 


Extensive: very 

expensive 

software, 

hardware 

requirements, 

slow process, 

learning curve, 

requires 

sysadmin. 


Moderate to 
Extensive: more 
powerful 
software with 
commercial 
support; requires 
a sysadmin for 
installation. 


Moderate: 
simplistic 
software, no 
commercial 
support, requires 
sysadmin. 


Moderate: 
simplistic 
software, no 
commercial 
support, requires 
sysadmin. 


Moderate: 
simplistic 
software, no 
commercial 
support, requires 
sysadmin. 


Extensive: 
requires 
sysadmin to 
install, 

knowledge of 
SAS; complex, 
powerful system, 
somewhat cryptic 
co debase, steep 
learning curve to 
SAS. 


Relative Power 


* * 


***** 


***** 


* * 


* 


* 


***** 


Installation 

Difficulty 


* * * 


* * * * 


***** 


* * 


* * 


* * 


***** 


Ease of use 134 


* * * 


* * * * 


* * * * 


* * 


* * 


* * 


* * 



133 Anonymization isn't explicitly impossible with these packages, but the feature is unique to our process. As the need hasn't been anticipated by other packages and the 
technique has unique properties (it affects reverse hostname lookup) the effect may be undesirable if used with other packages. 

134 The in-house software is very complex for performing simple analysis but appropriate for complex analysis. We've rated many packages as being "easy to use," but it is 
largely due to the very restricted set of functionality they offer (meaning it wouldn't be so simple to hack a simple program to perform complex analyses). 
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Appendix N: Online website survey instrument (from SPIRO site) 

1. Which title best describes you? 

University/ college instructor/ professor 

Undergraduate student 

Graduate student 

Independent researcher/ scholar 

Librarian 

K-12 instructor 

K-12 student 

Other 

2. For what purposes do you use the SPIRO site? (Check all that apply.) 

Conducting research 

Creating presentations, including lectures 
Developing teaching materials 
Making my own collection of digital resources 
Other. Please specify. 

3. How often do you use the SPIRO site? 

Today is my first time 

Daily 

Weekly 

Monthly 

Every six months 
Once a year 
Less than once a year 
Other. Please specify. 

4. Where are you accessing the site from now? 

Home 

Dormitory 

On campus office/ lab 

Library 

Office/ lab 

Other. Please specify. 

5. How often do you do the following activities on the SPIRO site? (Almost all the time; Often; 
Sometimes; Rarely; Never) 

Find text 
Download text 
Find images 

Download thumbnail images (128 pixels) 

Download larger images (650 pixels) 

Find images to license 

Browse to see what's available 

Use the site as a reference tool to verify citations 

Other. Please specify. 



http:/ / digitalresourcestudy.berkeley.edu 



10-111 




Use and Users of Digital Resources — Harley et al./ CSHE/UCB 



April 2006 



6. If you download larger images (650 pixels) from SPIRO, does this size meet your needs? 

Yes 

No 

Other. Please specify. 

7. What is your affiliation? 

Research university. Please specify. 

4-year college. Please specify. 

Community college. Please specify. 

High School 

Elementary or middle school 
Library. Please specify. 

Museum. Please specify. 

Governmental agency. Please specify. 

Business. Please specify. 

Other. Please specify. 
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Appendix O: CSHE Digital Resource Providers Study - Interview Protocol 

INTRODUCTION 

Thank you for agreeing to take part in this important study about user research. The interview should 
take about 1 hour. Your participation is voluntary, and you may decline to answer any questions or stop 
this interview at any time. 

I want to talk with you today about the <name ofproject> website. There are no right or wrong answers; 
we are simply interested in your opinion and your experiences. Please listen to each question carefully 
and answer as accurately as possible. 

DEMOGRAPHICS 

1) Name 

2) Title 

3) In what capacity have you been affiliated with the <name of project> website? (prompt: what 
is your role?) 

4) How long have you been affiliated with the <name of project> website? 

5) It's my understanding that the <name of project> website is <content/purpose/description 
summary>. Is this correct? (Could you briefly describe the type of website that you currently operate? 
Wltat IS the site?) 

FOCUS OF THE COEEECTION 

6) I'm going to read a list of possible users that your website serves. I will then ask you if that 
particular user base is currently one of your actual audiences, and if that user base was one of 
your originally intended audiences. 

Do K-12 students currently use your website? [ | 

Were K-12 students an originally intended audience for your website? QJ 

Do K-12 teachers currently use your website? ( | 

Were K-12 teachers an originally intended audience for your website? ' 
Do college/university students currently use your website? Q 

Were college/university students an originally intended audience for your website? | | 

Do college/university teachers currently use your website? ( | 

Were college/university teachers an originally intended audience for your website? [ | 

Do scholars currently use your website? Q 

Were scholars an originally intended audience for your website? [ j 

Does the general public currently use your website? Q 
Was the general public an originally intended audience for your website? [ j 
Are there any other users (not mentioned already) that currently use your website? | j 

Explain: 

Were any other users (not mentioned already) an originally intended audience for your 

website? j j 

Explain: 
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If there are no differences between actual and intended audiences, skip to Question 6c. 

6b) How have you responded to these new audiences in your planning, design, budgeting, 
etc.? Has this actual audience influenced your site's goals, content, or services? 

6c) Are there costs to accommodating secondary and/ or "informal learners"? 

6d) Is it important for you to distinguish among various educational audiences and contexts 
(e.g., community college, Rls, AP courses, fully online vs. on-campus hybrid, international 
markets, liberal arts, vocational, etc.)? 

6e) How important is it to distinguish between those willing to pay and not pay for use of 
online resources (e.g., undergraduate contexts vs. informal learners, etc.)? Between intentional 
users who have specific educational goals and broad spectrum users? 

COLLECTION GOALS 

7) For what purpose or use are your materials intended? Wltat are the goals of your site? (Skip to 
question 10 if educational purposes are mentioned) 

8) Are there specific goals/ intentions for using materials from the <name of project> website in 
formal educational contexts? Informal ? Please explain. 

9) Are you aware of any possible educational uses for the <name of project> website? 

|_j Yes 

□ No 

MEASURING USE 

The following questions address issues around measuring use and users. We would like to 
better understand the relationship between the information about users that you seek to 
understand, the data you collect (and the methods you use to collect it), and how you apply that 
knowledge. 

10a) What are the most important questions about users for which you want or need 
answers? 

10b) What metrics do you use/have you used to answer those questions? 

10c) Now I'd like to talk a little about the types of data you collect to get at those specific 
metrics. I'm going to read a list of possible ways to collect data about website use. Please 
answer "yes" if you use this method to measure the usage of your website, or answer "no" if 
you do not use the method. 

[ 1 do not do any user research at this time 

□ user registration Why? 

□ analysis of web traffic logs Why? 

[ j online surveys Why? 

i~~| help questions - analyzed Why? 

[_] email feedback Why? 
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□ 


analysis of user search strategies 


□ 


anecdotal research Why? 


□ 


interviews 


Why? 


□ 


focus groups 


Why? 


□ 


usability tests 


Why? 


□ 


other 


Why? 



Why? 



12a) Which data are most successful at answering the questions you have about users? What 
collected information has been useful? What has not been useful? 

12b) How does this information influence the website? Of these methods, have you used the 
results? (Connection between research and decision-making.) 

11) Now I'm going to read a list of possible reasons why information about users may be 
collected. Please answer "yes" or "no" if the listed reason explains why you collect usage data 
in general. 

j i curiosity 

j ] defend site funding 

j | appeal for more funding 

□ use for development (assess server size, etc.) 

I 1 target marketing efforts/ increase use by new audiences. Explain (replaces Q. 18) 
j j amend content 

j | change design 

| j other 

13) Can you give me a general overview of what you currently know about the usage of the 

<name of project> website? 

14) What do you know about what users do with the materials / information they retrieve? 
(Prompt: how collection materials are used in teaching & learning, types of activities, etc.) 

15) Are you satisfied with the level of use of your site? If not, what would you like to see 
change? 

16) What would you like to know about the <name of project> website usage that you have not 
been able to determine? 

17) What barriers make it difficult for you to conduct all of the user research that you might like 



to do? 




□ 


time 


□ 


staff 


u 


funding 


□ 


knowledge or skills 


□ 


technical challenges 


□ 


low priority relative to other tasks 


u 


other 



Nozv I'd like to ask you a few questions about user research in general. 
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18a) In your opinion, where are the gaps in current data about users, and how might those 
gaps be filled? What are the best methods to employ for strategic planning purposes? 

18b) Should or can online (or open) projects be thinking about common questions, metrics, 
and approaches to understanding users? 

18c) In your opinion, what are the consequences, if any, of poor sampling? Application of 
user results to wrong questions? Are there consequences to not doing user demand/ market 
analyses before building content? 

HISTORY AND DEVELOPMENT OF THE COLLECTION 

19) How did the collection originate? Did you. . . 
r/1 Digitize existing print material 

M License existing digital material from external sources 
| | Organize your own existing digital material 

| j Create new digital material 

I | Other 

20) Is content contributed to the site by outside users (non-staff)? 

Lj Yes 

L J No (skip to question 20) 

21) Is there a formal review process for the materials that are contributed? 

LJ Yes 

□ No 

Explain 

22) How often is the collection updated? 

j ) Never 

I | Daily 

I | Weekly 

□ Twice a month 

LJ Monthly 

j ■ Every few months 

□ 6 months 

[J Annually 



23) Are you aware of existing/ similar materials available? 

Where? 

Which resources? 

24) What year was the site first launched? 

25) How do you market the <name of project> website to potential users? 

[ J Personal contacts 

j 1 Advertising 

| j Other sites link to your site 

| | Search engine optimization (designing the site explicitly in a way that it will improve ranking 

for search engine results) 
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I | Other 

MANAGEMENT AND FUNDING 

26) Approximately how many staff members work on the <name of project> website? 

Actual 

FTE 

27) What percentage of your budget (staff time) is dedicated to assessing website use? 

28) What is the <name of project> website's current operating budget? 

□ no budget 

□ <10K 

fi 10-49K 

M 50-99K 

□ 100-499K 

Lj 500-1 MIL 

| i >1MIL 

29) I'm going to read a list of possible funding methods. Please indicate "yes" if the <name of 
project> website is currently funded using this method, or "no" if it is not. 

L^j self-supporting (through registration or licensing income) 

[ 1 self-supporting (through advertising on the site) 

I I supported by income from the rest of the organization 

| | private funding 

LJ grants from foundations 

1 1 government grants 

M other 

30) Is the current funding model sustainable? 

□ Yes (explain, then skip to question 32) 

□ No 

Explain 

31) What are your plans to make it sustainable? 

(Have you developed a business plan, sustainability model? If so, when did you develop this model? If 
not, do you have plans to develop one? Anticipated future funding sources from list?) 

P ! self-supporting (through registration or licensing income) 

Qj self-supporting (through advertising on the site) 

| j supported by income from the rest of the organization 

| j private funding 

| | grants from foundations 

i j government grants 

M other 

31b) How do you both give something away and at the same time create a revenue model that 
will allow it to support itself? 
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42) What role does "success" play in the sustainability of your project? 

43a) How do you know if your website is successful or valuable? 

43b) Who or what determines that value? 

43c) How is that value measured? Hoiv many users do you need to have to be considered successful? 
Fundable? How would your site persist if you had fewer than that number (or hardly any) ? 

44a) What kind of plan is in place, if any, to ensure the ongoing technical sustainability of 
your website? (e.g., software/ hardware upgrades and support, archiving) 

44b) What about content updates? 

44c) Has your institution (or organization) made any type of formal commitment to support 
your website? 

32) I'm going to read a list of possible funding methods. Please answer "yes" if the <name of 

project> website was originally financed using the listed funding method, or "no" if it was not. 
| I self-supporting (through registration or licensing income) 

I j self-supporting (through advertising on the site) 

[_J supported by income from the rest of the organization 

j i private funding 

[ 1 grants from foundations 

I | government grants 

| | other 

33) Who provided the funding support in the past? 

34) What was the site/ collection's approximate start-up budget? 



□ 


no budget 


□ 


<10K 


□ 


10-49K 


□ 


50-99K 


□ 


100-499K 


□ 


500-1 MIL 


□ 


>1MIL 



TECHNICAL QUESTIONS ABOUT THE SITE 

The following questions are intended to help us better understand how you make the content of your site 
useable to various audiences or learning communities. 

35) Are the items on the site stored in some sort of database? 

LJ Yes 

LI No (skip to question 38) 

Explain 

36) Does the database use some sort of structured metadata (a structured description of each 
item with separate fields, such as title, author, date, subject, etc.)? 
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M Yes 

□ No 

Explain 

37) Is the database and/ or metadata scheme designed to be interoperable with other systems; 
for example, so that your collection could be merged with other similar collections? 

I | Yes 

□ No 

Explain 

38) Is the collection currently included as part of a merged collection, or included in any cross- 
collection searching? Prompt: Have you provided your materials to any other groups who want to 
distribute your material on their website? 

I | Yes 

□ No 

Explain 

39) Does the collection include any materials imported (or licensed) from another collection or 
source? Prompt: Did you obtain any of your digital materials from some other website or collection ? 

M Yes 

□ No 

Explain 

40) Are the materials on the site specifically designed to be exportable into another system (e.g., 
a learning management system)? 

I | Yes 

□ No 

Explain 

Before we conclude the interview, I'd like to give you the opportunity to address any issues that we may 
have not covered. 

41) Is there anything else that you'd like to tell us? (Prompt: please feel free to comment about 
any topic related to the <name of project> website. Is there anything that else that we have not 
discussed that may be useful to know about the <name of project> website?) 

Follow-up Questions 135 

45) Please provide the percentage breakdown of use for your site during a normal month 
according to these categories: 

K-12 students: 

K-12 teachers: 

College/ University students: 

College/ University teachers: 

Scholars (university-based researchers): 



135 Sent via email to individual participants 
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General Public: 

Other (please describe if you know who these users are): 

46) Please provide your total usage during a normal month (use metrics that are most beneficial 
to you. ..hits, unique visitors, etc.) 
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Appendix P: OER Meeting - Participants 
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Appendix Q: OER Meeting - Agenda 

Meeting held at the Center for Studies in Higher Education, UC Berkeley, May 23-24, 2005 

ONLINE EDUCATIONAL RESOURCES: WHY STUDY USERS? 

What do we know about users of Online Educational Resources (OER)? What more do we need 

to know, and how do we find out? 

• Our target is to begin consolidating knowledge about the spectrum of current and 
potential users of online educational resources (OER), and their motivations for using 
these resources. Topics for discussion will include why and how educational content 
providers conduct user studies, what they actually know about their users, how that 
knowledge is applied in strategic planning, and how research results can be shared more 
easily. 

• The format will be relatively informal and flexible to maximize discussion. Proceedings 
will be recorded, transcribed, and summarized. 



MONDAY, MAY 23, 2005 

9:00 - 9:30 Continental breakfast available: Geballe Room, Townsend Center, 220 
Stephens Hall 

9:30 - 11:15 Introductions & Perspectives: Who are the current and potential users of OER? 
How do we know? 

What do developers know about the demographics, behavior, and motivations of users and 
non-users of online educational resources? How do they find out? What do they want to 
know? 

Survey of select projects (~10 minutes each) 

• Curricula (CMU, Connexions, Merlot, UC Irvine, MIT) 

• Digital Libraries (RLG, JSTOR, NSDL) 

• Tools and Reuse (Carnegie, MIT, IKSME) 

• Foundation and Society perspectives (Fisher, Wang) 

11:15 - 12:30 Codifying content and contexts 

Diane Harley, Jonathan Henke, Gary Matkin 

Quick (30 min) overview CSHE findings on humanities / social science users in three higher 
education contexts (survey and focus group data, "authentic" use, non-users, barriers to use). 
Followed by discussion, critique, and recommendations. 

• How does the variety of online educational content available (canned curricula, digital 
libraries, media sites, electronic textbooks, learning object repositories, etc.) and the 
infinite number of ways users can combine resources (e.g., individual digital objects and 
canned curricula; mixture of free and proprietary content, comprehensive and niche 
resources) influence our ability to make sense of the current and future landscape of user 
behavior and motivation? 
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12:30 - 1:30 pm Lunch 

1:30 - 3:30 pm Making OER useful to varied users and communities 

Gary Matkin, Vijay Kumar, Torn Iiyoshi (comments followed by general discussion) 

• Is it important for content providers to distinguish among various educational 
audiences and contexts (e.g., community college, Rls, AP courses, fully online vs. on- 
campus hybrid, international markets, liberal arts, vocational, etc.)? 

• How is localization, translation of open content to be handled? Who are the mediators? 
How can we help users access and share pedagogical knowledge and innovations to 
better use OER? 

• How important is it to distinguish between those willing to pay and not pay for use of 
online resources (e.g., undergraduate contexts vs informal learners, etc.)? Between 
intentional users who have specific educational goals and broad spectrum users? 

3:30 - 5:30 pm Users, user demand, and sustainability 

Saul Fisher, Vijay Kumar, Roger Schonfeld, Candace Thille (comments followed by general discussion) 

• What do we mean by sustainability? What are the dimensions of sustainability for Open 
Content? 

• What is the relationship, if any, between understanding current and potential users, and 
questions of user demand and sustainability of open educational content? 

• Comparison of specific projects on sustainability models. What are the key aspects of a 
project that render it sustainable? 

6:00 pm Dinner 



TUESDAY, MAY 24, 2005 

8:30 - 10:30 am What do we want to know about users? How do we find out? 

Shannon Lawrence, Diane Harley, Flora McMartin, Roger Schonfeld 

Overview of CSHE interviews on OER user studies (SL, ~30 minutes followed by discussion of 

summary document to be made available prior to meeting) 

• Where are the gaps in current data about users, and how might those gaps be filled? 
What are the best methods to employ for strategic planning purposes? Should/ can OER 
projects be thinking about common questions, metrics, and approaches to 
understanding users? 

• What are consequences, if any, of poor sampling? Application of user results to wrong 
questions? Are there consequences to not doing user demand/ market analyses before 
building content? 

10:30 am - 12:00 pm Next Steps 

• What are best ways to leverage activities across multiple projects? 

• What are the larger research questions and agendas that need to be addressed? 

12:00 - 1:00 pm Adjournment & Lunch 
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